systemd

Chapter 13: systemd Deep Dive

systemd is the init system and service manager of modern Linux distributions, running as PID 1 to bootstrap userspace, manage service lifecycles, collect logs, and control cgroup resources. This chapter moves from architecture principles through unit file authoring, service management, timers, log queries, and performance analysis — culminating in a complete production-grade systemd config for a Go web service.

1. systemd Architecture and Unit Types

systemd runs as PID 1 — the very first userspace process after the kernel boots. It starts services in parallel once their dependencies are satisfied, dramatically reducing boot time. systemd uses units as its management objects, each unit type representing a category of resource or behavior:

Dependency Keywords

• Requires= — Hard dependency: if dependency fails, this unit fails too
• Wants= — Soft dependency: failure does not stop this unit
• After= — Ordering: start after the specified unit
• Before= — Ordering: start before the specified unit
• BindsTo= — Strong binding: if dependency stops, this unit stops
• Conflicts= — Mutual exclusion: starting this unit stops the conflicting one

Target Concept

Targets are synchronization points for units, analogous to SysV runlevels. Common targets: multi-user.target (multi-user without GUI, equivalent to runlevel 3) and graphical.target (desktop environment, runlevel 5). default.target is the system's boot target, usually symlinked to graphical.target.

2. systemctl Command Reference

Service Lifecycle

# 启动 / 停止 / 重启 / 优雅重载配置（不重启进程）
systemctl start   nginx.service
systemctl stop    nginx.service
systemctl restart nginx.service
systemctl reload  nginx.service   # 发送 SIGHUP，要求服务支持

# 开机自启 / 禁用 / 屏蔽（mask 防止任何方式启动）
systemctl enable  nginx.service
systemctl disable nginx.service
systemctl mask    nginx.service
systemctl unmask  nginx.service

# 查看状态（含最近日志）
systemctl status nginx.service

# 重新读取磁盘上的 unit 文件（改完配置必须执行）
systemctl daemon-reload

State Queries

# 退出码 0=active / 非0=inactive
systemctl is-active  nginx.service
systemctl is-enabled nginx.service
systemctl is-failed  nginx.service

# 列出所有已加载的 unit
systemctl list-units
systemctl list-units --type=service --state=running

# 列出所有 unit 文件及其 enable 状态
systemctl list-unit-files --type=service

# 查看某 unit 的所有依赖树
systemctl list-dependencies nginx.service

Tip: In scripts, use systemctl is-active --quiet nginx — quiet mode returns only an exit code without output, making it easy to use in conditionals.

3. Writing .service Unit Files

System-level unit files live in /etc/systemd/system/ (highest priority) or /lib/systemd/system/ (distribution-provided). Custom services always belong in /etc/systemd/system/.

[Unit] Section: Description and Dependencies

[Unit]
Description=My Application Service
Documentation=https://example.com/docs
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service] Section: Process Control

Type= controls how systemd determines the service is "ready":

• simple — default; ExecStart process is the main process
• forking — service forks and parent exits (traditional daemon)
• oneshot — runs a one-time task and exits
• notify — service calls sd_notify() to signal readiness
• exec — like simple, but waits until exec() completes

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/myapp

# 环境变量（直接写或从文件读）
Environment=APP_ENV=production
Environment=PORT=8080
EnvironmentFile=/etc/myapp/env   # 文件中每行 KEY=VALUE

ExecStart=/opt/myapp/bin/server
ExecStop=/bin/kill -s TERM $MAINPID
ExecReload=/bin/kill -s HUP $MAINPID

# 自动重启策略
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=60s
StartLimitBurst=3

# 资源限制
LimitNOFILE=65536
LimitNPROC=4096

[Install] Section: Installation Target

[Install]
WantedBy=multi-user.target

Note: After modifying a unit file you must run systemctl daemon-reload, otherwise systemd continues using the in-memory old config even if the file changed.

4. .timer Units: Replacing cron

A systemd timer requires a paired .service unit with the same base name. When the timer fires, systemd starts the corresponding service.

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily database backup timer

[Timer]
# 系统启动后 5 分钟执行一次
OnBootSec=5min
# 每次触发后 24 小时再次触发
OnUnitActiveSec=24h
# 或者使用 cron 风格（每天 03:00）
OnCalendar=*-*-* 03:00:00
# 系统关机期间错过的任务，开机后补执行
Persistent=true
# 随机延迟，避免整点流量洪峰
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Database backup job

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/backup.sh

# 启用并启动 timer（注意启用的是 timer 而不是 service）
systemctl enable --now backup.timer

# 查看所有 timer 及下次触发时间
systemctl list-timers --all

crontab vs systemd timer Comparison

5. .socket Units: Socket Activation

Socket activation is one of systemd's core features: systemd pre-creates and holds the socket, only starting the corresponding service when a client connects, achieving zero listen latency and faster boot.

# /etc/systemd/system/myapp.socket
[Unit]
Description=MyApp TCP socket

[Socket]
ListenStream=0.0.0.0:8080
# Unix domain socket 示例
# ListenStream=/run/myapp.sock
# 数据报 socket（UDP）
# ListenDatagram=514

# socket 传给服务后保持监听（多实例场景）
Accept=false

[Install]
WantedBy=sockets.target

How it works: The service receives the socket via file descriptor inheritance (starting at SD_LISTEN_FDS_START=3) without calling bind()/listen() itself. In Go, use net.FileListener to create a Listener from the inherited fd.

6. .path Units: Filesystem-Triggered Activation

# /etc/systemd/system/process-uploads.path
[Unit]
Description=Watch upload directory for new files

[Path]
# 路径存在时触发（首次检测）
PathExists=/var/spool/uploads/trigger
# 路径内容变化时触发（inotify IN_CLOSE_WRITE）
PathChanged=/var/spool/uploads
# 路径被修改时触发（包括属性变化）
PathModified=/var/spool/uploads
# 触发后启动同名 .service（或用 Unit= 指定其他服务）
Unit=process-uploads.service

[Install]
WantedBy=multi-user.target

7. journalctl Complete Guide

journald is systemd's built-in structured logging system, replacing traditional syslog. All service stdout/stderr goes into the journal automatically — no log file path configuration required.

# 查看某个服务的日志（-u 过滤 unit）
journalctl -u nginx.service

# 实时跟踪（类似 tail -f）
journalctl -u nginx.service -f

# 时间范围过滤
journalctl -u nginx --since "2025-01-01 00:00:00" --until "2025-01-02 00:00:00"
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since yesterday

# 按优先级过滤（emerg/alert/crit/err/warning/notice/info/debug）
journalctl -p err           # 只看 err 及更高级别
journalctl -p warning..err  # 范围过滤

# 当前启动的日志
journalctl -b
journalctl -b -1            # 上次启动
journalctl --list-boots     # 列出所有启动记录

# 输出格式
journalctl -u nginx -o json        # JSON 格式，每条一行
journalctl -u nginx -o json-pretty # 格式化 JSON
journalctl -u nginx -o verbose     # 所有字段
journalctl -u nginx -o short-iso   # ISO 时间戳

# 日志磁盘占用与清理
journalctl --disk-usage
journalctl --vacuum-size=500M      # 清理到 500MB
journalctl --vacuum-time=30d       # 清理 30 天以前的日志

# 按进程/用户过滤
journalctl _PID=1234
journalctl _UID=1000

# 内核日志
journalctl -k                      # 等同 dmesg

Production tip: Persist journal logs: by default they live in /run/log/journal/ (lost on reboot). Create /var/log/journal/ and restart journald to persist logs to /var/log/journal/.

8. cgroup v2 Resource Control

systemd creates an individual cgroup for each service. Resource limits can be set directly in the [Service] section of the unit file without manually touching /sys/fs/cgroup.

[Service]
# CPU 配额：最多使用 2 个核心的时间（200% 表示 2 核）
CPUQuota=200%
# CPU 权重（相对调度优先级，默认 100，范围 1-10000）
CPUWeight=200

# 内存限制
MemoryMax=512M       # 硬上限，超过触发 OOM kill
MemoryHigh=400M      # 软上限，超过开始节流
MemorySwapMax=0      # 禁止使用 swap

# IO 权重（相对，默认 100）
IOWeight=50

# 任务（线程）数限制
TasksMax=128

# 实时监控各 cgroup 资源占用（类似 top）
systemd-cgtop

# 查看某服务的 cgroup 路径
systemctl status nginx | grep "CGroup"

# 直接查看 cgroup v2 层级树
ls /sys/fs/cgroup/system.slice/nginx.service/

# 查看某 cgroup 的 CPU 统计
cat /sys/fs/cgroup/system.slice/nginx.service/cpu.stat

9. systemd-analyze: Boot Performance

# 总体启动时间分解
systemd-analyze

# 各 unit 启动耗时排行（最慢的在最上面）
systemd-analyze blame

# 关键路径（决定最终启动时间的依赖链）
systemd-analyze critical-chain

# 生成启动时序 SVG 图（可在浏览器打开）
systemd-analyze plot > boot.svg

# 检查 unit 文件语法
systemd-analyze verify /etc/systemd/system/myapp.service

# 分析特定 unit 的安全评分
systemd-analyze security nginx.service

Optimization tip: After finding bottlenecks with systemd-analyze blame, check whether you can: ① change After=network.target to After=network-online.target for precise network readiness, ② use WantedBy instead of Requires for non-critical services, ③ use socket activation to defer service startup.

10. User-level systemd

Each logged-in user can run their own systemd instance to manage user-level services without root. Unit files go in ~/.config/systemd/user/.

# 用户级 systemctl（加 --user 标志）
systemctl --user start  myservice
systemctl --user enable myservice
systemctl --user status myservice

# 查看用户级 journal
journalctl --user -u myservice -f

# 允许服务在用户注销后继续运行（默认注销时停止）
loginctl enable-linger $USER

# 用户级 unit 文件路径
mkdir -p ~/.config/systemd/user/
# 放置 unit 后需要 reload
systemctl --user daemon-reload

11. Practice: Complete systemd Config for a Go Web Service

The following is a production-ready systemd configuration for a Go web service, including security hardening, automatic restart, resource limits, and log configuration.

# /etc/systemd/system/goapp.service
[Unit]
Description=Go Web Application Server
Documentation=https://github.com/myorg/goapp
After=network-online.target postgresql.service redis.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=notify
User=goapp
Group=goapp
WorkingDirectory=/opt/goapp

# 二进制和配置
ExecStart=/opt/goapp/bin/goapp serve --config /etc/goapp/config.yaml
ExecReload=/bin/kill -s HUP $MAINPID

# 环境变量
Environment=GOMAXPROCS=4
EnvironmentFile=-/etc/goapp/env   # - 前缀表示文件不存在时不报错

# 自动重启：非0退出码或信号退出时重启，最多5分钟内3次
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=300s
StartLimitBurst=3

# 安全加固
NoNewPrivileges=yes
PrivateTmp=yes            # 独立 /tmp 目录
ProtectSystem=strict      # 只读挂载 /usr /boot /etc
ProtectHome=yes           # 无法读取家目录
ReadWritePaths=/var/lib/goapp /var/log/goapp
CapabilityBoundingSet=CAP_NET_BIND_SERVICE  # 只允许绑定低端口
AmbientCapabilities=CAP_NET_BIND_SERVICE

# 资源限制
LimitNOFILE=65536
CPUQuota=400%
MemoryMax=1G
MemoryHigh=800M
TasksMax=256

# 标准输出直接进 journal
StandardOutput=journal
StandardError=journal
SyslogIdentifier=goapp

[Install]
WantedBy=multi-user.target

Companion timer: scheduled log cleanup

# /etc/systemd/system/goapp-cleanup.timer
[Unit]
Description=GoApp log cleanup timer

[Timer]
OnCalendar=Sun *-*-* 02:00:00
Persistent=true
RandomizedDelaySec=600

[Install]
WantedBy=timers.target

---
# /etc/systemd/system/goapp-cleanup.service
[Unit]
Description=GoApp log cleanup

[Service]
Type=oneshot
User=goapp
ExecStart=/opt/goapp/scripts/cleanup-logs.sh

# 部署步骤
sudo cp goapp.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now goapp.service

# 验证状态
sudo systemctl status goapp.service
sudo journalctl -u goapp.service -f

# 查看服务的 cgroup 资源占用
sudo systemd-cgtop -d 1

Security score: Run systemd-analyze security goapp.service to get a sandbox security score (0-10, higher is safer). The config above with NoNewPrivileges/PrivateTmp/ProtectSystem/ProtectHome should achieve around 7+.

  Previous
  ← Ch12: Engineering


  Next
  Ch14: Performance →