Logs: /var/log, journalctl and logrotate

Logs are the first thing to read in any incident, and one of the commonest causes of one: a log file that grows without limit fills the disk and stops the application. This post covers where logs come from, how to read them quickly, and how to rotate and retain them.

Versions in use on this date. rsyslog 8.32 and logrotate 3.11 on Ubuntu 18.04.3 LTS. CentOS 7.7 uses rsyslog 8.24 and logrotate 3.8. Both systems also run systemd-journald, which captures service output as well.

Where logs come from

flowchart TB
  K[Kernel messages] --> J[systemd-journald]
  SV[systemd services<br/>stdout and stderr] --> J
  APP[Apps using syslog calls] --> J
  J --> RS[rsyslog]
  RS --> F1[var/log/syslog or messages]
  RS --> F2[var/log/auth.log or secure]
  APP2[Apps writing own files<br/>nginx mysql] --> F3[var/log/nginx/*.log]
  F1 --> LR[logrotate runs daily from cron or timer]
  F2 --> LR
  F3 --> LR
  LR --> AR[Rotated and compressed archives]
  AR --> DEL[Deleted after retention]

Files to know

Ubuntu	CentOS	Contents
/var/log/syslog	/var/log/messages	General system messages
/var/log/auth.log	/var/log/secure	Logins, sudo, SSH
/var/log/kern.log, dmesg	dmesg	Kernel, hardware, OOM kills
/var/log/apt/	/var/log/yum.log	Package changes
/var/log/nginx/, /var/log/mysql/	same	Application logs
/var/log/cron.log (if enabled)	/var/log/cron	Scheduled job runs

Reading logs quickly

sudo tail -f /var/log/syslog
sudo less +F /var/log/nginx/error.log          # follow inside less; Ctrl-C then / to search
sudo grep -i "error" /var/log/syslog | tail -50
sudo zgrep "Failed password" /var/log/auth.log.*.gz
journalctl -u nginx --since "30 min ago"
journalctl -p warning -b                        # warnings and worse since boot
journalctl _PID=1234
dmesg -T | tail -30

Work backwards from the time of the incident. Note timestamps and time zones, and check several logs around that minute (app, web server, auth, kernel).

logrotate

logrotate renames the current file, compresses old ones and deletes the oldest. It runs daily (/etc/cron.daily/logrotate). Per-application rules live in /etc/logrotate.d/:

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

Directive	Meaning
daily / weekly / size 100M	When to rotate
rotate 14	Keep 14 old files
compress, delaycompress	gzip old files, but leave the latest rotated copy uncompressed
create mode user group	New empty file after rotation
copytruncate	Copy then truncate in place, for programs that cannot reopen files (can lose lines)
postrotate	Tell the application to reopen its log (a reload or kill -USR1)

Without the postrotate step, a program keeps writing to the renamed file and disk space is not freed.

sudo logrotate -d /etc/logrotate.d/myapp       # debug: show what would happen
sudo logrotate -f /etc/logrotate.d/myapp       # force a rotation now
cat /var/lib/logrotate/status                  # last rotation times (Ubuntu)

The journal’s size

journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=14d

Make limits permanent in /etc/systemd/journald.conf (SystemMaxUse=500M), then sudo systemctl restart systemd-journald.

When the disk is full of logs

sudo du -ah /var/log | sort -rh | head
sudo lsof +L1 | grep deleted          # deleted files still held open by a process
sudo truncate -s 0 /var/log/big.log   # safe way to empty a log being written

Deleting a log that a process still has open does not free space until the process closes or restarts; truncate instead.

Beyond one server

On a fleet, ship logs to a central place (rsyslog forwarding, or an agent to an ELK or CloudWatch-style service) so evidence survives a failed server. Keep authentication logs for the period your security policy requires, set alerts on error spikes, and never log passwords or tokens.