Debugging with strace, lsof and tcpdump

When logs say nothing and the application just hangs or fails, ask the operating system what the process is actually doing. Three tools answer most of those questions: strace shows the system calls a process makes, lsof shows which files and sockets it has open, and tcpdump shows the packets on the wire.

Versions in use on this date. strace 4.21, lsof 4.89 and tcpdump 4.9.3 on Ubuntu 18.04.4 LTS. Install with sudo apt install strace lsof tcpdump. All three need root (or the right capabilities) to attach to other users’ processes.

Which tool for which question

flowchart TB
  Q[Something is wrong with a process] --> A{What kind of problem?}
  A -->|Hangs or slow| ST[strace -f -tt -T -p PID<br/>which call blocks]
  A -->|Permission or missing file| ST2[strace -e trace=file<br/>see exact path and errno]
  A -->|Disk full or locked file| LS[lsof<br/>who has it open]
  A -->|Port in use or connection leak| LS2[lsof -i and ss -tp]
  A -->|Cannot reach remote service| TD[tcpdump<br/>do packets leave and return]

strace: system calls

sudo strace -f -tt -T -o /tmp/trace.txt -p 1234         # attach to a running process
strace -f -e trace=file ls /nonexistent                 # only file-related calls
strace -c -p 1234                                       # summary: time per syscall (Ctrl-C to stop)
strace -e trace=network -f curl -s https://example.com -o /dev/null

Option	Meaning
`-p PID`	Attach to a running process
`-f`	Follow child processes and threads
`-tt -T`	Timestamps and time spent in each call
`-e trace=file,network,process`	Filter classes of calls
`-s 200`	Show longer strings
`-o file`	Write output to a file

Reading the output. openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = -1 ENOENT (No such file or directory) tells you exactly which path the program looked for and failed to find. A process stuck on futex(...) is waiting on a lock; stuck in read(5, ...) or poll(...) is waiting for I/O on file descriptor 5, which you can identify with ls -l /proc/1234/fd/5.

strace slows the target noticeably, so use it briefly on production processes and prefer -c or filtered traces.

lsof: open files and sockets

sudo lsof -p 1234                      # everything this process has open
sudo lsof /var/log/syslog              # who has this file open
sudo lsof -i :80                       # who uses port 80
sudo lsof -i -P -n | grep ESTABLISHED  # network connections with numbers, not names
sudo lsof +L1                          # deleted files still held open
sudo lsof -u www-data | wc -l          # descriptor count for a user
sudo lsof +D /mnt/data                 # blocks unmount: who is using this directory

Real cases:

Disk full but du shows free space: a deleted log is still open. lsof +L1 lists it; restart the process or truncate through /proc/PID/fd/N.
umount: target is busy: lsof +D /mnt/data or fuser -vm /mnt/data.
“Too many open files”: compare lsof -p PID | wc -l with ulimit -n and the unit’s LimitNOFILE.

tcpdump: packets

sudo tcpdump -i any -nn host 10.0.0.5 and port 5432
sudo tcpdump -i eth0 -nn -c 20 port 53                       # DNS queries and answers
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'     # connection setup and resets
sudo tcpdump -i any -nn -s0 -w /tmp/capture.pcap port 443    # save for Wireshark

Option	Meaning
`-i any`	All interfaces
`-nn`	No name or port resolution (faster, clearer)
`-c N`	Stop after N packets
`-w file`	Write a pcap file; read with `-r` or Wireshark
`-A` / `-X`	Print payload as ASCII / hex

Reading the result. You see SYN and no SYN-ACK: a firewall or routing drops it. You see RST straight after SYN: nothing listens on that port. Retransmissions: a lossy path. Traffic to the wrong DNS server or IP: configuration. HTTPS payloads are encrypted, so use tcpdump to prove connectivity and timing, not content.

Capture only what you need (filters, short counts), because captures may contain passwords and personal data; delete them after use and follow your data policy.

A worked example

A service says “database connection timed out”. Check, in order:

ss -tnp | grep 5432: is a connection attempt stuck in SYN-SENT?
sudo tcpdump -i any -nn host 10.0.0.9 and port 5432 -c 10: SYN packets leave and nothing returns.
Conclusion: a firewall or security group blocks it, or the database host is down. Ask the network owner with evidence (timestamps and the capture) rather than guessing.