Understand Top Command Output

Top Command: As a Linux system administrator, top command is a frequently used command and our daily tasks we used multiple times this commands to view resource utilization by processes on server. This command helps us to find which process is utilizing what resources of system.

Using this article, I am trying to write to how to use and read results of top command.

Result Row #1:

Row 1 results shows about server up time from last reboot, currently logged in users and cpu load on server. The same output you can find using linux uptime command.

Result Row #2:

Row 2 shows the number of process running on server and there state.

Zombie process or defunct process is a process that has completed execution but still has an entry in the process table. This entry is still needed to allow the parent process to read its child’s exit status.

Result Row #3:

Row three shows the cpu utilization status on server, you can find here how much cpu is free and how much is utilizing by system.

Definition from access.redhat.com:
Steal time is the time that a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.

Result Row #4:

Row 4 shows the memory utilization on server, you can find here how much memory is used, the same results you can find using free command.

Result Row #5:

Row 4 shows the swap memory utilization on server, you can find here how much swap is being used, the same results you can find using free command.

Result Row #6 ( Running Processes ):

In this steps you will see all running process on servers and there additional details about them like below.

If you want to sort output on basis of any fields, you can use following key combinations.

Press "SHIFT + F" and Select your choice below and press ENTER.

  A     ( PID = Process Id )
  b     (  PPID   = Parent Process Pid )
  d     (  UID= User Id )
  e     (  USER   = User Name )
  f     (  GROUP  = Group Name ) 
  g     (  TTY= Controlling Tty )
  h     (  PR = Priority )
  i     (  NI = Nice value )  
  j     (  P  = Last used cpu (SMP) )
  k     (  %CPU  = CPU usage )
  l     (  TIME  = CPU Time )
  n     (  %MEM  = Memory usage (RES) )
  o     (  VIRT  = Virtual Image (kb) )
  p     (  SWAP  = Swapped size (kb) )
  q     (  RES= Resident size (kb) )
  r     (  CODE  = Code size  (kb) )
  s     (  DATA  = Data+Stack size (kb) )
  t     (  SHR= Shared Mem size (kb) )
  w     (  S  = Process Status )
  x     (  COMMAND= Command name/line )

-- Srikanth Govada

Script to gather DEBUG data for SUN web server when HUNG in SOLARIS

#wshung.ksh

#!/bin/ksh -p

# Copyright 2008 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
# Script for Gathering Debug Data for Web Server 6.0, 6.1 and 7.0 (and Proxy
# Server 4.0) on Solaris SPARC and x86
#
# Version: 1.3 - unknown
# 1.4 - Feb 2007 - verified support for 7.0, added netstat
# 2.0 - Apr 2007 - combo script incl. support for pre-Solaris 8,
# optional gcore
# 2.1 - May 2008 - Integrated pkg_app, command line pid entry, pid
# check, extra data collection, add Proxy 4.0
# 3.0 - Nov 2008 - Integrate pkgapp v3, modified command line
# pid check, added truss, modified proc tool order
# 3.1 - Mar 2009 - bug fix to command line pid check
#
# Check for a new version of this script:
# http://www.sun.com/bigadmin/scripts/indexSjs.html
# Obtain the Sun GDD documentation and/or check for new product coverage:
# http://www.sun.com/service/gdd/index.xml
#
# Please send all script BUGs and/or RFEs to the following address:
# Subject 'wshang bug/rfe' - gdd-issue-tracker@sun.com
# Please send all script BUGs and/or RFEs to the following address:
# Subject 'wshang feedback' - gdd-feedback@sun.com

#
# USAGE: Run the script with no arguments to see all Web processes running on
# the machine.
#

# The following four settings can be changed as necessary:

# Change this value to set the duration in secs between data collection rounds
DURATION=15

# Set the following variable to y or n to allow a gcore to be taken
CORECHOICE=y

# Set the following variable to y or n to allow the pkgapp script to run.
# Assumes the pkgapp script is in the same location as wshang script.
PKGAPP=y

# Set the following variable with your Case Number. If no case number has been
# opened but wish to use wshang.ksh to trap an issue, you can use your Company
# Name (no spaces) or all 1's. Once you get a Sun Case number, you can rename
# the resulting tar file before sending to Sun.
CASENUMBER=3-9395377381

############################################################
# Work starts here - no need to change anything below here #
############################################################

# prompt for response for different directory
resp=$1

# Ensure /usr/bin is in the path
PATH=/usr/bin:$PATH

# add the date to the results file
TIMESTAMP=`date "+%y%m%d-%H%M%S"`

# Check what version of Solaris is in use
SOLVER=$(uname -r | cut -c3-4)

echo "
This script collects 3 snapshots of the following information
at a ${DURATION} seconds interval against a web server process:
pstack
pfiles
prstat -L -a (will try 'top' on pre-Solaris 8 systems)
pflags
pmap -x
pldd
netstat -an
And collections 30 seconds of:
truss -feald -vall -wall -rall
";

# Check to see if user wants to place data in different location
echo "Data will be collected to the location /tmp/wshang by default."
echo "Data location should have enough disk space for large core files."
echo "Would you like to specify a different directory for data? "
echo "(y/n) "
read resp

if [ $resp = "y" ]; then
TMPDIR=$2
echo "Specify a directory: "
read TMPDIR
pid=$3
else
# The temporary directory where the results will be saved
TMPDIR=/tmp/wshang
pid=$2
fi

if [ -d $TMPDIR ]; then
echo " "
echo "$TMPDIR exists, may I delete it? (y/n)"
echo "WARNING: If your response is "y", $TMPDIR will be deleted!"
read resp
if [ $resp = "y" ]; then
rm -r -f $TMPDIR/*
fi
fi

# create the directory for data collection
mkdir -p $TMPDIR

#
# Mar 2009: Start interactive user inputs
#

echo " "
echo " "
echo "Enter your Sun case id or service request number: "
read CASENUMBER

# check if input was given
if [ -z $CASENUMBER ] ; then
CASENUMBER=11111111
fi

echo " "
echo " "
echo "Do you wish to collect a core file? (y/n) Default is yes."
read CORECHOICE

# check if input was given
if [ "$CORECHOICE" != "y" -a "$CORECHOICE" != "n" ]; then
CORECHOICE="y"
fi

echo " "
echo " "
echo "How many seconds do you wish to configure between proc tool data collection? Default is $DURATION seconds"
read DURATION

# check if input was given
if [ -z $DURATION ]; then
DURATION=15
fi

echo " "
echo $CORECHOICE

if [ "$CORECHOICE" == "y" ]; then
echo " "
echo " "
echo "The pkgapp is used by Sun to open your core file for analysis."
echo "Do you wish to collect pkgapp data? (y/n) Default is $PKGAPP "
read PKGAPP

# check if input was given
if [ "$PKGAPP" != "y" -a "$PKGAPP" != "n" ]; then
PKGAPP="y"
fi
fi

#
# If pid was not entered on the cmd line then list the running web server
# instances and prompt user to choose
#

if [ -z "$pid" ]; then

echo " "
echo " "
echo "The following is a list of web server processes:"
echo " "

# If Solaris 8 or greater
if (( $SOLVER > 7 )) ; then
echo "Solaris 8 or greater"
echo "Web Server 7.0/6.1 processes"
#ptree | grep ' webservd ' | awk '((NR % 2) == 1 ) {print}' | sed 1d
ptree | grep '^ [0-9]' | grep ' webservd '

echo "Web Server 6.0 processes"
#ptree | grep ns-httpd | awk '((NR % 2) == 1) {print}' | sed 1d
ptree | grep '^ [0-9]' | grep ' ns-httpd '

echo "Proxy Server 4.0 processes"
#ptree | grep ' proxyd ' | awk '((NR % 2) == 1) {print}' | sed 1d
ptree | grep '^ [0-9]' | grep ' proxyd '

else
echo "Solaris 7 or less"
# Less elegant. Note Proxy 4.x not supported < Solaris 8 ps -ef | grep -i https- fi echo " " echo " " echo "Enter the PID of the web server process to collect data on" echo "(Note that the correct PID for child webservd/ns-httpd process will not be listed correctly here if you have MaxProcs > 1 in your magnus.conf. Please manually find the pid you need to collect data against.):"
read pid

# End pid test
fi

# Check to see if the pid actually exists
ps -p $pid
if [ $? -ne 0 ]; then
echo "ERROR: Specified pid ($pid) does not exist"
exit 1
fi

# If Solaris less than 8
if (( $SOLVER < 8 )) ; then echo "Adding /usr/proc/bin to the path so the tools work" PATH=$PATH:/usr/proc/bin fi echo " " echo " " echo "Gathering data now..." echo " " # # Collect the information # Includes conditional check for Solaris 8 and greater to run prstat # for i in 1 2 3 do echo "Loop $i for $DURATION seconds..." echo " " echo "collecting pflags..." echo "*******************************" >> $TMPDIR/pflags.out.$i
date >> $TMPDIR/pflags.out.$i
pflags $pid >> $TMPDIR/pflags.out.$i

echo "collecting pmap -x..."
echo "*******************************" >> $TMPDIR/pmap.out.$i
date >> $TMPDIR/pmap.out.$i
pmap -x $pid >> $TMPDIR/pmap.out.$i

echo "collecting pldd..."
echo "*******************************" >> $TMPDIR/pldd.out.$i
date >> $TMPDIR/pldd.out.$i
pldd $pid >> $TMPDIR/pldd.out.$i

if (( $SOLVER > 7 )) ; then
echo "collecting prstat -L -a..."
echo "*******************************" >> $TMPDIR/prstat.out.$i
date >> $TMPDIR/prstat.out.$i
prstat -L -a -p $pid 1 1 >> $TMPDIR/prstat.out.$i
else
echo "collecting top..."
echo "*******************************" >> $TMPDIR/top.out.$i
date >> $TMPDIR/top.out.$i
top -d 1 -n 20 >> $TMPDIR/top.out.$i
fi

echo "collecting pstack..."
echo "*******************************" >> $TMPDIR/pstack.out.$i
date >> $TMPDIR/pstack.out.$i
pstack $pid >> $TMPDIR/pstack.out.$i

echo "collecting pfiles..."
echo "*******************************" >> $TMPDIR/pfiles.out.$i
date >> $TMPDIR/pfiles.out.$i
pfiles $pid >> $TMPDIR/pfiles.out.$i

echo "collecting netstat..."
echo "*******************************" >> $TMPDIR/netstat.out.$i
date >> $TMPDIR/netstat.out.$i
netstat -an >> $TMPDIR/netstat.out.$i

echo " "

# last round no need to sleep
if [ $i -eq 3 ] ; then
echo "collecting 30 seconds of truss..."
truss -feald -o $TMPDIR/truss.out -vall -wall -rall -p $pid &
sleep 30
pkill -9 truss
echo "done"
echo " "
break
fi

# sleep
sleep $DURATION
done

# In addition gcore the process based on variable set in script

if [ $CORECHOICE = "y" ] ; then
echo "Collecting gcore..."
gcore -o $TMPDIR/gcore.out $pid

# If Solaris 8 or greater only as p commands are only for pid
# on earlier versions
if (( $SOLVER > 7 )) ; then
pstack $TMPDIR/gcore.out.$pid > $TMPDIR/gcore_pstack.out
pmap $TMPDIR/gcore.out.$pid > $TMPDIR/gcore_pmap.out
pldd $TMPDIR/gcore.out.$pid > $TMPDIR/gcore_pldd.out

# Check if a pkgapp script run is required
if [ $PKGAPP = "y" ] ; then

# Firstly get path to the webservd binary file for use by pkgapp

# This will do 7.0
SERVER_ROOT=`pmap $pid | grep webservd | grep lib`

# This will do 6.1
if [ -z "$SERVER_ROOT" ]; then
SERVER_ROOT=`pmap $pid | grep webservd | grep bin`
fi

# Need a check for WS 6.0 and ns-httpd
if [ -z "$SERVER_ROOT" ]; then
SERVER_ROOT=`pmap $pid | grep ns-httpd | grep bin/https/bin`
fi

# This will do Proxy 4.0
if [ -z "$SERVER_ROOT" ]; then
SERVER_ROOT=`pmap $pid | grep proxyd | grep bin/proxy/bin`
fi

SERVER_ROOT=`echo $SERVER_ROOT | cut -f4 -d' '`
echo $SERVER_ROOT

# usage:./pkgapp -c -p ./pkgapp -c $pid -p $SERVER_ROOT -s $TMPDIR/pkgapp_data
fi
fi

else
echo "No gcore requested"
fi

# End of gcore section

# Collect some other useful data

showrev -p > $TMPDIR/showrev.out
uname -a > $TMPDIR/uname.out
cat /etc/release > $TMPDIR/release.out
ulimit -a > $TMPDIR/ulimit

# Gather everything up
echo "Creating a tar file of the results..."
cd $TMPDIR 1>/dev/null 2>&1
tar cvfh ${CASENUMBER}-gdd-wshang-${TIMESTAMP}.tar *

# Try and gzip the result - might not be installed on Solaris 7 and 2.6
gzip ${CASENUMBER}-gdd-wshang-${TIMESTAMP}.tar

echo " "
echo "The tar file has been created...."
echo " "
echo "Please send the file ${TMPDIR}/${CASENUMBER}-gdd-wshang-${TIMESTAMP}.tar.gz to"
echo "Sun Microsystems at https://supportfiles.sun.com/upload"
echo "Choose cores as the destination."
echo " "

# Automatically change permissions if required (uncomment this)
# chmod 777 ${TMPDIR}/${CASENUMBER}-gdd-wshang-${TIMESTAMP}.tar.gz

# end



#explore advanced UNIX techniques to troubleshoot performance and instance hung problems.

pstack_truss_doc



-- Srikanth Govada