Managing Services with systemd

systemd is the init system on Ubuntu 18.04 and CentOS 7. It starts services at boot in dependency order, supervises them, restarts them when they fail and collects their logs. A few commands and one unit file cover most day-to-day work.

Versions in use on this date. systemd 237 on Ubuntu 18.04.3 LTS and systemd 219 on CentOS 7.6. The older version lacks some newer directives (for example some sandboxing options), so test unit files on both if you run both.

How a service starts

flowchart TB
  B[Boot reaches a target<br/>multi-user.target] --> W[Wants and Requires<br/>pull in units]
  W --> O[After and Before<br/>set start order]
  O --> U[myapp.service]
  U --> EP[ExecStartPre checks]
  EP --> ES[ExecStart runs main process]
  ES --> JD[stdout and stderr to journald]
  ES --> F{Process exits?}
  F -->|unexpected| RS[Restart policy waits RestartSec]
  RS --> ES
  F -->|stopped by admin| ST[inactive]

Everyday commands

sudo systemctl status nginx          # state, PID, recent log lines
sudo systemctl start|stop|restart|reload nginx
sudo systemctl enable --now nginx    # start at boot and start now
sudo systemctl disable nginx
systemctl is-active nginx; systemctl is-enabled nginx
systemctl list-units --type=service --state=failed
systemctl list-unit-files --state=enabled
systemctl cat nginx                  # the unit file plus drop-ins
systemctl list-dependencies nginx

reload asks the service to re-read config without dropping connections (when supported); restart stops and starts it. Run nginx -t (or the equivalent config test) before either.

Logs with journalctl

journalctl -u nginx                       # one unit
journalctl -u nginx -f                    # follow
journalctl -u nginx --since "1 hour ago"
journalctl -p err -b                      # errors since this boot
journalctl -b -1                          # previous boot (needs persistent journal)
journalctl --disk-usage
sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald    # make logs persistent

On Ubuntu 18.04 the journal is already persistent if /var/log/journal exists; CentOS 7 keeps it in memory unless you create the directory.

Write your own unit

Unit files for your own software go in /etc/systemd/system/, never /usr/lib/systemd/system/ (owned by packages).

# /etc/systemd/system/myapp.service
[Unit]
Description=My Node.js API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/usr/bin/node server.js
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload            # required after any unit change
sudo systemctl enable --now myapp
sudo systemctl status myapp
journalctl -u myapp -f

Notes: use absolute paths in ExecStart; keep secrets in the EnvironmentFile with chmod 600; Restart=on-failure restarts crashes but not a clean systemctl stop. systemd-analyze verify /etc/systemd/system/myapp.service checks syntax.

Change vendor units with drop-ins

sudo systemctl edit nginx           # creates /etc/systemd/system/nginx.service.d/override.conf

[Service]
LimitNOFILE=100000
Restart=always

Drop-ins survive package upgrades, whereas editing the packaged file does not.

Targets and boot problems

systemctl get-default                # multi-user.target (server) or graphical.target
systemd-analyze blame | head         # slow boot units
systemd-analyze critical-chain
sudo systemctl rescue                # single-user mode

Troubleshooting a failing service

systemctl status myapp and journalctl -xe -u myapp for the last error.
Run the ExecStart command by hand as the service user (sudo -u myapp ...) to reproduce.
Check Result= and exit code in systemctl show myapp -p ExecMainStatus -p Result.
start-limit-hit means it crashed too often: fix the cause, then systemctl reset-failed myapp.
Remember services do not read your shell profile; set environment variables in the unit.