Processes: ps, top and Signals

Every running program is a process with a PID, an owner, a state and a parent. Troubleshooting a slow or stuck server mostly means answering three questions: what is running, what is using resources, and how do I stop it cleanly.

Versions in use on this date. Ubuntu 18.04.2 LTS with procps-ng 3.3.12 (the package that provides ps, top, free, pgrep and kill). CentOS 7.6 ships procps-ng 3.3.10 and behaves the same for everything below.

The life of a process

flowchart TB
  P[Parent calls fork] --> C[Child process created]
  C --> X[exec loads a new program]
  X --> R[Running or runnable R]
  R --> S[Sleeping S<br/>waiting for I/O or events]
  R --> D[Uninterruptible D<br/>stuck in disk or NFS I/O]
  R --> T[Stopped T<br/>Ctrl-Z or SIGSTOP]
  R --> Z[Zombie Z<br/>exited, parent has not collected]
  Z --> W[Parent calls wait and entry is removed]

Looking at processes

ps aux                              # every process, BSD style
ps -ef --forest                     # tree view with parent relationships
ps -eo pid,ppid,user,%cpu,%mem,stat,etime,cmd --sort=-%mem | head
pgrep -a nginx                      # find by name
pstree -p | head

top is the live view. Useful keys: P sort by CPU, M by memory, 1 per-CPU lines, c full command, k kill, q quit. htop (from the repositories) is friendlier.

Read the header of top before the list:

Field	Meaning
load average	Runnable plus waiting-on-disk tasks over 1, 5, 15 minutes; compare with CPU count (nproc)
%Cpu us / sy / wa / st	User, kernel, I/O wait, stolen by hypervisor
KiB Mem: free / buff/cache	Linux uses spare RAM for cache, so low “free” is normal; check “avail Mem”
S column	State letter from the diagram above

High wa points to disk; high st on a cloud VM means a noisy neighbour; many D processes mean storage problems.

Signals: asking a process to stop

kill -l                      # list signals
kill 1234                    # SIGTERM (15): please shut down cleanly
kill -HUP 1234               # SIGHUP (1): reload config (nginx, sshd, rsyslog)
kill -INT 1234               # SIGINT (2): same as Ctrl-C
kill -9 1234                 # SIGKILL (9): cannot be caught or cleaned up
pkill -f 'python app.py'     # by pattern

The right order: send SIGTERM and wait a few seconds so the program can flush data and remove lock files; use SIGKILL only if it ignores that. SIGKILL cannot be caught, so databases may need recovery afterwards. A zombie cannot be killed (it is already dead); fix or restart its parent.

Background jobs and survival after logout

sleep 300 &                  # background
jobs; fg %1; bg %1           # job control; Ctrl-Z suspends
nohup ./long-task.sh > task.log 2>&1 &       # survives logout
nice -n 10 ./batch.sh        # lower priority
renice +10 -p 1234

For anything that must keep running, use a systemd service (see the systemd post), or tmux/screen for interactive work, instead of leaving jobs attached to your session.

Memory pressure and the OOM killer

free -h
dmesg -T | grep -i 'killed process'        # out-of-memory kills
cat /proc/1234/status | grep -E 'VmRSS|Threads'
ls -l /proc/1234/fd | wc -l                # open file descriptors

When RAM and swap run out, the kernel kills a process chosen by an OOM score. If the victim is your database, the fix is memory limits and capacity, not restarting.

Quick triage checklist

1. uptime for load, then top sorted by CPU and by memory.
2. df -h and free -h for disk and memory.
3. dmesg -T | tail for kernel complaints.
4. ps -eo stat,pid,cmd | awk '$1 ~ /^[DZ]/' for stuck and zombie processes.