Wednesday, April 2, 2014

Performance Monitoring Tools ( Unix/Linux commands )- Server Performance Stats


Monitoring the servers during performance testing is very important and useful to find out the actual bottleneck on any specific server, now a day's most of our applications runs on Non-windows machines ( Unix /Linux , etc..) hence we should know few commands to quick monitoring , I felt below are the bare minimum commands to monitor the performance of server.

By using Perfmon we can monitor WINDOW's based servers but not Non-Windows based machines, here are few, also if we have a facility of having Monitoring tools like CA APM wily , Site scope . Dyna trace , etc.... in our testing environment that's good , it will reduce this overhead.  But if you don't have that facility below commands will surely help.

Unix/Linux Commands for Performance Monitoring:

  • vmstat
  • sar
  • iostat
  • mpstat
  • free
  • top
  • netstat
  • ps


vmstat :


vmstat reports virtual memory statistics. The following are some of vmstat command examples.

$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----                                                                                        -
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0    332 1587232 106808 4368288    0    0   266 11974    1    7 12  6 78  5    

if you use vmstat 60 120  ,  120 reading will be taken for every 60 secs ( so for 2 hrs load test you can use this)

if you want have vmstat stats with time stamp use below command.


vmstat 3 5 | sed "s/$/$(date)/"

$ vmstat 3 5 | sed "s/$/$(date)/"
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------Wed Mar 26 14:32:28 UTC 2014
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa stWed Mar 26 14:32:28 UTC 2014
 2  0    332  44600 118912 6070464    0    0   266 11974    1    7 12  6 78  5  0Wed Mar 26 14:32:28 UTC 2014
 1  0    332  44020 119104 6061464    0    0    24 107865 2775 5262 16  7 74  4  0Wed Mar 26 14:32:28 UTC 2014


Get the "id" values (-----cpu------ Column) which is ideal time of CPU , 100-id value will give the CPU utilization.

From Above example : id value is 78 and 74, so 22 and 26 are CPU utilizations.

Sar


Using sar utility you can do two things: 1) Monitor system real time performance (CPU, Memory, I/O, etc) 2) Collect performance data in the background on an on-going basis and do analysis on the historical data to identify bottlenecks.

Sar is part of the sysstat package:

Example :

 sar 1 1
Linux 2.6.18-371.4.1.el5 (HostName)    03/26/2014

02:36:08 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
02:36:09 PM       all     17.48      0.00      6.24      5.12      0.00     71.16
Average:          all     17.48      0.00      6.24      5.12      0.00     71.16


iostat:


iostat reports CPU, disk I/O, and NFS statistics

Iostat without any argument showes CPU usage, and I/O statistics about all the partitions on the system as shown below:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.84    0.00    5.79    5.01    0.00   77.36

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
cciss/c0d0       16.41        52.06       365.14   59021735  413998136
cciss/c0d0p1      0.00         0.00         0.00       5262         16
cciss/c0d0p2     16.41        52.05       365.14   59016129  413998120
sda             156.52       704.20     19325.03  798415632 21910654896
sda1            156.52       704.19     19325.03  798404688 21910654896


To execute every 5 seconds for a total of 10 times, do the following

iostat 5 10


mpstat:



By default mpstat displays CPU statistics as shown below:

11:56:10 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
11:56:10 AM  all   11.84    0.00    4.44    5.01    0.06    1.29    0.00   77.36   2913.01


$ mpstat -P ALL

Option -P ALL, displays all the individual CPUs (or Cores) along with its statistics as shown below.

1:55:16 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
11:55:16 AM  all   11.84    0.00    4.44    5.01    0.06    1.29    0.00   77.36   2913.03
11:55:16 AM    0   12.49    0.00    3.96    3.52    0.00    1.31    0.00   78.72   1985.54
11:55:16 AM    1    9.64    0.00    4.73    9.93    0.24    4.80    0.00   70.65    496.52



To display statistics information of a particular CPU (or core) than use below
$ mpstat -P 3 ( means for 3rd core CPU stats ).

TOP:


This command will show us, which proccess is consuming more CPU and memory , this will be usefull to quickly check.

top - 12:08:17 up 13 days,  3:12,  5 users,  load average: 2.24, 1.94, 1.86
Tasks: 342 total,   3 running, 336 sleeping,   3 stopped,   0 zombie
Cpu(s): 15.9%us,  4.6%sy,  0.0%ni, 70.3%id,  7.4%wa,  0.1%hi,  1.7%si,  0.0%st
Mem:   8176252k total,  8132220k used,    44032k free,    87712k buffers
Swap:  4194288k total,      332k used,  4193956k free,  5941952k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16502 wasadm    23   0  390m 189m  21m S 96.3  2.4   1:03.30 java
10856 root      10  -5     0    0    0 S 15.5  0.0   2119:55 kmirrord
 8860 wasadm    14  -1 68724 1532  340 R  5.3  0.0   0:38.07 rsync
16363 wasadm    14  -1 70520 1564  340 S  5.0  0.0   0:03.01 rsync
18964 wasadm    14  -1 68468 1508  336 S  5.0  0.0   0:00.65 rsync

If you want to re-direct the output of any of these commands use  "  > filename ".

Example :

vmstat 60 120 > Sampleyloadtest.txt   ( Sampleyloadtest.txt file will be created on current directory and output will be saved for future analysis ).

Netstat:



The netstat command is a Command Prompt command used to display very detailed information about how your computer is communicating with other computers or network devices.

Specifically, the netstat command can show details about individual network connections, overall and protocol-specific networking statistics, and much more, all of which could help troubleshoot certain kinds of networking issues

For example:

netstat -an | head -10  -> will give top 10 connection details


netstat -an | grep "CONNECTED"  -> will give all connected status network details


PS

ps command will give us which process running on that server , will would help to find out process details on server.

most useful command is

ps -aef | grep java  [ will give all java process running on that server ].



No comments:

Post a Comment