{"id":1235,"date":"2012-03-24T15:51:52","date_gmt":"2012-03-24T15:51:52","guid":{"rendered":"https:\/\/noi3.org\/site\/?p=1235"},"modified":"2012-03-24T15:51:52","modified_gmt":"2012-03-24T15:51:52","slug":"20-linux-system-monitoring-tools-every-sysadmin-should-know1","status":"publish","type":"post","link":"https:\/\/site.noi3.org\/?p=1235","title":{"rendered":"20 Linux System Monitoring Tools Every SysAdmin Should Know"},"content":{"rendered":"
\t\"\"<\/div>\n
\t <\/div>\n

\tNeed to monitor Linux server performance? Try these built-in command and a few add-on tools. Most Linux distributions are equipped with tons of monitoring. These tools provide metrics which can be used to get information about system activities. You can use these tools to find the possible causes of a performance problem.<\/p>\n

<\/p>\n

\tNeed to monitor Linux server performance? Try these built-in command and a few add-on tools. Most Linux distributions are equipped with tons of monitoring. These tools provide metrics which can be used to get information about system activities. You can use these tools to find the possible causes of a performance problem. The commands discussed below are some of the most basic commands when it comes to system analysis and debugging server issues such as:
\t <\/p>\n

    \n
  1. \t\tFinding out bottlenecks.<\/li>\n
  2. \t\tDisk (storage) bottlenecks.<\/li>\n
  3. \t\tCPU and memory bottlenecks.<\/li>\n
  4. \t\tNetwork bottlenecks.<\/li>\n<\/ol>\n

    \t <\/p>\n

    \t#1: top – Process Activity Command<\/h2>\n

    \tThe top program provides a dynamic real-time view of a running system i.e. actual process activity. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.
    \t <\/p>\n

    \t\"Fig.01:<\/a>Fig.01: Linux top command
    \t <\/div>\n

    \tCommonly Used Hot Keys<\/h3>\n

    \tThe top command provides several useful hot keys:
    \t <\/p>\n\n\n\n\n\n\n\n\n\n\n\n
    \t\t\t\tHot Key<\/th>\n \t\t\t\tUsage<\/th>\n<\/tr>\n
    \t\t\t\tt<\/td>\n \t\t\t\tDisplays summary information off and on.<\/td>\n<\/tr>\n
    \t\t\t\tm<\/td>\n \t\t\t\tDisplays memory information off and on.<\/td>\n<\/tr>\n
    \t\t\t\tA<\/td>\n \t\t\t\tSorts the display by top consumers of various system resources. Useful for quick identification of performance-hungry tasks on a system.<\/td>\n<\/tr>\n
    \t\t\t\tf<\/td>\n \t\t\t\tEnters an interactive configuration screen for top. Helpful for setting up top for a specific task.<\/td>\n<\/tr>\n
    \t\t\t\to<\/td>\n \t\t\t\tEnables you to interactively select the ordering within top.<\/td>\n<\/tr>\n
    \t\t\t\tr<\/td>\n \t\t\t\tIssues renice command.<\/td>\n<\/tr>\n
    \t\t\t\tk<\/td>\n \t\t\t\tIssues kill command.<\/td>\n<\/tr>\n
    \t\t\t\tz<\/td>\n \t\t\t\tTurn on or off color\/mono<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    <\/p>\n

    \t=> Related:<\/span><\/strong> How do I Find Out Linux CPU Utilization?<\/a><\/p>\n

    \t<\/a><\/p>\n

    \t <\/p>\n

    \t#2: vmstat – System Activity, Hardware and System Information<\/h2>\n

    \tThe command vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.
    \t# vmstat 3<\/code>
    \tSample Outputs:<\/p>\n

    \tprocs ––––memory–––- –swap– ––io–- –system– ––cpu–– r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 2540988 522188 5130400 0 0 2 32 4 2 4 1 96 0 0 1 0 0 2540988 522188 5130400 0 0 0 720 1199 665 1 0 99 0 0 0 0 0 2540956 522188 5130400 0 0 0 0 1151 1569 4 1 95 0 0 0 0 0 2540956 522188 5130500 0 0 0 6 1117 439 1 0 99 0 0 0 0 0 2540940 522188 5130512 0 0 0 536 1189 932 1 0 98 0 0 0 0 0 2538444 522188 5130588 0 0 0 0 1187 1417 4 1 96 0 0 0 0 0 2490060 522188 5130640 0 0 0 18 1253 1123 5 1 94 0 0<\/p>\n

    \tDisplay Memory Utilization Slabinfo<\/h3>\n

    \t# vmstat -m<\/code><\/p>\n

    \tGet Information About Active \/ Inactive Memory Pages<\/h3>\n

    \t# vmstat -a<\/code>
    \t=> Related:<\/span><\/strong>     How do I find out Linux Resource utilization to detect system bottlenecks?<\/a><\/p>\n

    \t<\/a><\/p>\n

    \t#3: w – Find Out Who Is Logged on And What They Are Doing<\/h2>\n

    \tw command displays information about the users currently on the machine, and their processes.
    \t# w username
    \t# w vivek<\/code>
    \tSample Outputs:<\/p>\n

    \t17:58:47 up 5 days, 20:28, 2 users, load average: 0.36, 0.26, 0.24 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts\/0 10.1.3.145 14:55 5.00s 0.04s 0.02s vim \/etc\/resolv.conf root pts\/1 10.1.3.145 17:43 0.00s 0.03s 0.00s w<\/a>
    \t <\/p>\n

    \t#4: uptime – Tell How Long The System Has Been Running<\/h3>\n

    \tThe uptime command can be used to see how long the server has been running. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
    \t# uptime<\/code>
    \tOutput:<\/p>\n

    \t18:02:41 up 41 days, 23:42, 1 user, load average: 0.00, 0.00, 0.001 can be considered as optimal load value. The load can change from system to system. For a single CPU system 1 – 3 and SMP systems 6-10 load value might be acceptable.<\/p>\n

    \t<\/a>
    \t <\/p>\n

    \t#5: ps – Displays The Processes<\/h2>\n

    \tps command will report a snapshot of the current processes. To select all processes use the -A or -e option:
    \t# ps -A<\/code>
    \tSample Outputs:<\/p>\n

    \tPID TTY TIME CMD 1 ? 00:00:02 init 2 ? 00:00:02 migration\/0 3 ? 00:00:01 ksoftirqd\/0 4 ? 00:00:00 watchdog\/0 5 ? 00:00:00 migration\/1 6 ? 00:00:15 ksoftirqd\/1 …. ….. 4881 ? 00:53:28 java 4885 tty1 00:00:00 mingetty 4886 tty2 00:00:00 mingetty 4887 tty3 00:00:00 mingetty 4888 tty4 00:00:00 mingetty 4891 tty5 00:00:00 mingetty 4892 tty6 00:00:00 mingetty 4893 ttyS1 00:00:00 agetty 12853 ? 00:00:00 cifsoplockd 12854 ? 00:00:00 cifsdnotifyd 14231 ? 00:10:34 lighttpd 14232 ? 00:00:00 php-cgi 54981 pts\/0 00:00:00 vim 55465 ? 00:00:00 php-cgi 55546 ? 00:00:00 bind9-snmp-stat 55704 pts\/1 00:00:00 psps is just like top but provides more information.
    \t <\/p>\n

    \tShow Long Format Output<\/h3>\n

    \t# ps -Al<\/code>
    \tTo turn on extra full mode (it will show command line arguments passed to process):
    \t# ps -AlF<\/code><\/p>\n

    \tTo See Threads ( LWP and NLWP)<\/h3>\n

    \t# ps -AlFH<\/code><\/p>\n

    \tTo See Threads After Processes<\/h3>\n

    \t# ps -AlLm<\/code><\/p>\n

    \tPrint All Process On The Server<\/h3>\n

    \t# ps ax
    \t# ps axu<\/code><\/p>\n

    \tPrint A Process Tree<\/h3>\n

    \t# ps -ejH
    \t# ps axjf
    \t# pstree<\/code><\/p>\n

    \tPrint Security Information<\/h3>\n

    \t# ps -eo euser,ruser,suser,fuser,f,comm,label
    \t# ps axZ
    \t# ps -eM<\/code><\/p>\n

    \tSee Every Process Running As User Vivek<\/h3>\n

    \t# ps -U vivek -u vivek u<\/code><\/p>\n

    \tSet Output In a User-Defined Format<\/h3>\n

    \t# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
    \t# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
    \t# ps -eopid,tt,user,fname,tmout,f,wchan<\/code><\/p>\n

    \tDisplay Only The Process IDs of Lighttpd<\/h3>\n

    \t# ps -C lighttpd -o pid=<\/code>
    \tOR
    \t# pgrep lighttpd<\/code>
    \tOR
    \t# pgrep -u vivek php-cgi<\/code><\/p>\n

    \tDisplay The Name of PID 55977<\/h3>\n

    \t# ps -p 55977 -o comm=<\/code><\/p>\n

    \tFind Out The Top 10 Memory Consuming Process<\/h3>\n

    \t# ps -auxf | sort -nr -k 4 | head -10<\/code><\/p>\n

    \tFind Out top 10 CPU Consuming Process<\/h3>\n

    \t# ps -auxf | sort -nr -k 3 | head -10<\/code><\/p>\n

    \t<\/a><\/p>\n

    \t#6: free – Memory Usage<\/h2>\n

    \tThe command free displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.
    \t# free <\/code>
    \tSample Output:<\/p>\n

    \ttotal used free shared buffers cached Mem: 12302896 9739664 2563232 0 523124 5154740 -\/+ buffers\/cache: 4061800 8241096 Swap: 1052248 0 1052248=> Related:<\/span><\/strong> :
    \t <\/p>\n

      \n
    1. \t\tLinux Find Out Virtual Memory PAGESIZE<\/a><\/li>\n
    2. \t\tLinux Limit CPU Usage Per Process<\/a><\/li>\n
    3. \t\tHow much RAM does my Ubuntu \/ Fedora Linux desktop PC have?<\/a><\/li>\n<\/ol>\n

      \t<\/a><\/p>\n

      \t#7: iostat – Average CPU Load, Disk Activity<\/h2>\n

      \tThe command iostat report Central Processing Unit (CPU) statistics and input\/output statistics for devices, partitions and network filesystems (NFS).
      \t# iostat <\/code>
      \tSample Outputs:<\/p>\n

      \tLinux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06\/26\/2009 avg-cpu: %user %nice %system %iowait %steal %idle 3.50 0.09 0.51 0.03 0.00 95.86 Device: tps Blk_read\/s Blk_wrtn\/s Blk_read Blk_wrtn sda 22.04 31.88 512.03 16193351 260102868 sda1 0.00 0.00 0.00 2166 180 sda2 22.04 31.87 512.03 16189010 260102688 sda3 0.00 0.00 0.00 1615 0=> Related:<\/span><\/strong> : Linux Track NFS Directory \/ Disk I\/O Stats<\/a><\/p>\n

      \t<\/a>
      \t <\/p>\n

      \t#8: sar – Collect and Report System Activity<\/h2>\n

      \tThe sar command is used to collect, report, and save system activity information. To see network counter, enter:
      \t# sar -n DEV | more<\/code>
      \tTo display the network counters from the 24th:
      \t# sar -n DEV -f \/var\/log\/sa\/sa24 | more<\/code>
      \tYou can also display real time usage using sar:
      \t# sar 4 5<\/code>
      \tSample Outputs:<\/p>\n

      \tLinux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06\/26\/2009 06:45:12 PM CPU %user %nice %system %iowait %steal %idle 06:45:16 PM all 2.00 0.00 0.22 0.00 0.00 97.78 06:45:20 PM all 2.07 0.00 0.38 0.03 0.00 97.52 06:45:24 PM all 0.94 0.00 0.28 0.00 0.00 98.78 06:45:28 PM all 1.56 0.00 0.22 0.00 0.00 98.22 06:45:32 PM all 3.53 0.00 0.25 0.03 0.00 96.19 Average: all 2.02 0.00 0.27 0.01 0.00 97.70=> Related:<\/span><\/strong> : How to collect Linux system utilization data into a file<\/a><\/p>\n

      \t<\/a>
      \t <\/p>\n

      \t#9: mpstat – Multiprocessor Usage<\/h2>\n

      \tThe mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:
      \t# mpstat -P ALL<\/code>
      \tSample Output:<\/p>\n

      \tLinux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06\/26\/2009 06:48:11 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr\/s 06:48:11 PM all 3.50 0.09 0.34 0.03 0.01 0.17 0.00 95.86 1218.04 06:48:11 PM 0 3.44 0.08 0.31 0.02 0.00 0.12 0.00 96.04 1000.31 06:48:11 PM 1 3.10 0.08 0.32 0.09 0.02 0.11 0.00 96.28 34.93 06:48:11 PM 2 4.16 0.11 0.36 0.02 0.00 0.11 0.00 95.25 0.00 06:48:11 PM 3 3.77 0.11 0.38 0.03 0.01 0.24 0.00 95.46 44.80 06:48:11 PM 4 2.96 0.07 0.29 0.04 0.02 0.10 0.00 96.52 25.91 06:48:11 PM 5 3.26 0.08 0.28 0.03 0.01 0.10 0.00 96.23 14.98 06:48:11 PM 6 4.00 0.10 0.34 0.01 0.00 0.13 0.00 95.42 3.75 06:48:11 PM 7 3.30 0.11 0.39 0.03 0.01 0.46 0.00 95.69 76.89=> Related:<\/span><\/strong> : Linux display each multiple SMP CPU processors utilization individually<\/a>.<\/p>\n

      \t<\/a>
      \t <\/p>\n

      \t#10: pmap – Process Memory Usage<\/h2>\n

      \tThe command pmap report memory map of a process. Use this command to find out causes of memory bottlenecks.
      \t# pmap -d PID<\/code>
      \tTo display process memory information for pid # 47394, enter:
      \t# pmap -d 47394<\/code>
      \tSample Outputs:<\/p>\n

      \t47394: \/usr\/bin\/php-cgi Address Kbytes Mode Offset Device Mapping 0000000000400000 2584 r-x– 0000000000000000 008:00002 php-cgi 0000000000886000 140 rw– 0000000000286000 008:00002 php-cgi 00000000008a9000 52 rw– 00000000008a9000 000:00000 [ anon ] 0000000000aa8000 76 rw– 00000000002a8000 008:00002 php-cgi 000000000f678000 1980 rw– 000000000f678000 000:00000 [ anon ] 000000314a600000 112 r-x– 0000000000000000 008:00002 ld-2.5.so 000000314a81b000 4 r–- 000000000001b000 008:00002 ld-2.5.so 000000314a81c000 4 rw– 000000000001c000 008:00002 ld-2.5.so 000000314aa00000 1328 r-x– 0000000000000000 008:00002 libc-2.5.so 000000314ab4c000 2048 –– 000000000014c000 008:00002 libc-2.5.so ….. …… .. 00002af8d48fd000 4 rw– 0000000000006000 008:00002 xsl.so 00002af8d490c000 40 r-x– 0000000000000000 008:00002 libnss_files-2.5.so 00002af8d4916000 2044 –– 000000000000a000 008:00002 libnss_files-2.5.so 00002af8d4b15000 4 r–- 0000000000009000 008:00002 libnss_files-2.5.so 00002af8d4b16000 4 rw– 000000000000a000 008:00002 libnss_files-2.5.so 00002af8d4b17000 768000 rw-s- 0000000000000000 000:00009 zero (deleted) 00007fffc95fe000 84 rw– 00007ffffffea000 000:00000 [ stack ] ffffffffff600000 8192 –– 0000000000000000 000:00000 [ anon ] mapped: 933712K writeable\/private: 4304K shared: 768000KThe last line is very important:
      \t <\/p>\n