Log Management with journalctl¶

Overview¶

Modern Linux distributions store most system and service logs in the systemd journal — a structured, indexed, binary log managed by systemd-journald. The journalctl command is the primary interface for querying, filtering, and exporting these logs.

Unlike flat text files in /var/log, the journal supports metadata such as unit names, priorities, boot IDs, and process IDs. This makes it far easier to answer production questions like "what failed after the last deploy?" or "show me all errors from nginx in the previous boot."

This tutorial is part of Module 6: Storage, Logs, Networking & Operations in the REBASH Academy Linux series. You will learn to filter by priority, follow logs in real time, inspect previous boots, configure persistent storage, and integrate journal data into your operational workflow.

Prerequisites¶

• Complete systemd Service Management or equivalent experience
• A Linux VM, cloud instance, or local machine running systemd (Ubuntu 20.04+, RHEL 8+, Debian 11+)
• sudo privileges for persistent journal configuration exercises
• SSH or console access to run lab commands

Learning Objectives¶

By the end of this tutorial, you will be able to:

• Explain how systemd-journald collects and stores log data
• Filter logs by priority, time range, systemd unit, and boot session
• Follow logs in real time and export entries for analysis or SIEM ingestion
• Configure persistent journal storage and retention limits
• Troubleshoot missing logs, permission issues, and disk pressure from journal growth

Architecture Diagram¶

flowchart TB
    subgraph Sources
        K[Kernel]
        S[systemd Units]
        A["Applications via stdout/stderr"]
    end

    subgraph journald
        J[systemd-journald]
        R["(Journal Files<br/>/run/log/journal<br/>/var/log/journal")]
    end

    subgraph Consumers
        C[journalctl CLI]
        F["Forward to SIEM/syslog"]
        M[Monitoring alerts]
    end

    K --> J
    S --> J
    A --> J
    J --> R
    R --> C
    R --> F
    R --> M```

## Theory

### The systemd journal

When a service writes to stdout or stderr, or when the kernel emits a message, `systemd-journald` captures it and stores it with metadata fields:

| Field | Meaning |
|-------|---------|
| `_SYSTEMD_UNIT` | Service unit name (e.g., `nginx.service`) |
| `PRIORITY` | Syslog severity level (0–7) |
| `_PID` | Process ID |
| `_BOOT_ID` | Unique identifier for each boot session |
| `_HOSTNAME` | Host that generated the entry |
| `MESSAGE` | Log message text |

Journal files live in `/run/log/journal/` (volatile, cleared on reboot) by default. When persistent storage is enabled, copies are also written to `/var/log/journal/`.

### Log priority levels

`journalctl` uses syslog priority names and numeric shortcuts:

| Priority | Name | Typical use |
|----------|------|-------------|
| 0 | emerg | System unusable |
| 1 | alert | Immediate action required |
| 2 | crit | Critical conditions |
| 3 | err | Error conditions |
| 4 | warning | Warning conditions |
| 5 | notice | Normal but significant |
| 6 | info | Informational |
| 7 | debug | Debug-level messages |

Filter with `-p err` (errors and above) or `-p warning..err` (range syntax).

### Boot sessions and unit filters

Each reboot creates a new boot ID. Use `-b` for the current boot, `-b -1` for the previous boot, and `-b 0` for all boots. Unit filters (`-u nginx.service`) narrow output to a single service — essential when debugging application failures.

### Persistent journal configuration

By default, many cloud images use volatile journals to save disk space. Production servers should enable persistence in `/etc/systemd/journald.conf`:

```ini
[Journal]
Storage=persistent
SystemMaxUse=500M
MaxRetentionSec=30day

After changes, restart systemd-journald with sudo systemctl restart systemd-journald.

journalctl vs traditional log files¶

Text logs in /var/log/syslog or /var/log/messages remain on some systems, but systemd-centric distros increasingly rely on the journal. Use journalctl for unified querying; use file-based logs when legacy tools or applications require them.

Hands-on Lab¶

Step 1 – View recent journal entries¶

Display the last 20 log lines from the current boot:

journalctl -n 20 --no-pager

Expected output (sample):

Jul 27 14:02:11 lab-server systemd[1]: Started Daily apt download activities.
Jul 27 14:02:12 lab-server sshd[1240]: Accepted publickey for ubuntu from 10.0.1.5
Jul 27 14:02:15 lab-server kernel: eth0: link up
...

Step 2 – Filter by systemd unit¶

Inspect logs for the SSH service (unit name varies by distribution):

journalctl -u ssh -n 10 --no-pager 2>/dev/null \
  || journalctl -u sshd -n 10 --no-pager

Expected output: Lines prefixed with the unit name showing connection attempts, key auth, or session open/close events.

Step 3 – Filter by priority and time¶

Show errors from the last hour, then all warnings and above:

journalctl --since "1 hour ago" -p err --no-pager | tail -5
journalctl -p warning..emerg -n 10 --no-pager

Expected output: Only entries at the specified severity; timestamps within your chosen window.

Step 4 – Follow logs in real time¶

Start live tailing (like tail -f), then generate activity in another terminal:

journalctl -f

In a second terminal:

logger "REBASH lab test message"

Expected output: The -f session prints the new line immediately with your hostname and the test message.

Press Ctrl+C to stop following.

Step 5 – Inspect previous boots¶

List available boots and view logs from the last reboot:

journalctl --list-boots
journalctl -b -1 -n 15 --no-pager

Expected output:

IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -1 abc123...                         Mon 2026-07-26 09:00:01     Mon 2026-07-27 08:59:58
  0 def456...                         Mon 2026-07-27 09:00:02     Mon 2026-07-27 14:30:00

The -b -1 output shows shutdown events, kernel messages, and service stops from the prior session.

Step 6 – JSON export for parsing¶

Export structured JSON for scripting or SIEM pipelines:

journalctl -u ssh -n 3 -o json --no-pager 2>/dev/null \
  || journalctl -u sshd -n 3 -o json --no-pager

Expected output: One JSON object per line with fields like "MESSAGE", "PRIORITY", "_SYSTEMD_UNIT".

Step 7 – Check journal disk usage¶

journalctl --disk-usage
sudo ls -lh /var/log/journal/ 2>/dev/null || echo "Persistent journal not enabled"

Expected output:

Archived and active journals take up 48.0M in the file system.

Step 8 – Enable persistent journal (optional, requires sudo)¶

sudo mkdir -p /var/log/journal
sudo sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
grep ^Storage= /etc/systemd/journald.conf
sudo systemctl restart systemd-journald
journalctl --disk-usage

Expected output: Storage=persistent in config; disk usage may increase after restart as logs are flushed to /var/log/journal/.

Commands¶

Code Examples¶

Service failure investigation script¶

#!/bin/bash
# investigate-service.sh — quick journal triage for a failed unit
set -euo pipefail

UNIT="${1:?Usage: $0 <unit.service>}"

echo "=== Status: $UNIT ==="
systemctl status "$UNIT" --no-pager -l || true

echo
echo "=== Last 30 journal lines ==="
journalctl -u "$UNIT" -n 30 --no-pager

echo
echo "=== Errors since last boot ==="
journalctl -u "$UNIT" -b -p err --no-pager

Export logs for a support ticket¶

#!/bin/bash
# export-logs.sh — bundle recent errors for offline review
set -euo pipefail

OUT="/tmp/journal-export-$(date +%Y%m%d-%H%M%S).txt"

{
  echo "Host: $(hostname)"
  echo "Date: $(date -Iseconds)"
  echo "=== Errors (current boot) ==="
  journalctl -b -p err --no-pager
  echo "=== Failed units ==="
  systemctl --failed --no-pager
} > "$OUT"

echo "Exported to $OUT"

Cron-friendly log vacuum (maintenance)¶

# Run weekly via root crontab to prevent journal bloat
0 3 * * 0 /usr/bin/journalctl --vacuum-size=500M

Common Mistakes¶

Best Practices¶

Troubleshooting¶

Summary¶

• The systemd journal stores structured, indexed logs with rich metadata for units, priorities, and boots.
• journalctl filters by -u (unit), -p (priority), -b (boot), and time ranges — the core production toolkit.
• Use -f for live tailing during deploys and incidents; use -o json for automation.
• Enable persistent storage and retention limits on production servers to preserve logs across reboots and control disk usage.
• Combine journal queries with systemctl status and centralized log forwarding for complete observability.

Interview Questions¶

1. Where does systemd store journal files, and what is the difference between volatile and persistent storage?

Sample answer: Volatile journals live in /run/log/journal/ and are cleared on reboot. Persistent journals are stored in /var/log/journal/ when Storage=persistent is set in /etc/systemd/journald.conf. Production servers use persistent storage for post-incident analysis.

2. How do you show only error-level messages from the current boot?

Sample answer: journalctl -b -p err --no-pager. The -p err filter includes emerg, alert, crit, and err priorities.

3. How do you view logs from the previous boot after a crash?

Sample answer: journalctl --list-boots to identify boot indices, then journalctl -b -1 for the previous session. This requires persistent journaling if the crash prevented clean shutdown flushing.

4. What is the equivalent of tail -f for systemd logs?

Sample answer: journalctl -f follows new entries. Add -u UNIT to follow a specific service.

5. How do you investigate why nginx failed to start?

Sample answer: Run systemctl status nginx, then journalctl -u nginx -b -n 50 --no-pager. Look for configuration syntax errors, port conflicts, or permission issues in the output.

6. How can you reduce journal disk usage without disabling logging?

Sample answer: Use journalctl --vacuum-size=500M or --vacuum-time=30d, and set SystemMaxUse in journald.conf for automatic limits.

7. Why might an application log appear in /var/log/app.log but not in journalctl?

Sample answer: The application writes directly to a file instead of stdout/stderr. Only processes connected to systemd's standard streams or using sd_journal API appear in the journal unless configured otherwise.

8. What output format would you use to ingest logs into a SIEM?

Sample answer: JSON via journalctl -o json or json-pretty, optionally with time and unit filters. Many forwarders (Fluent Bit, Filebeat with journald input) read the journal natively.

9. What does journalctl -xe show, and when is it useful?

Sample answer: Recent entries with additional explanatory text from systemd — helpful when a service fails and systemd prints suggested diagnostic commands.

10. How do priority ranges work in journalctl?

Sample answer: -p warning..err shows warnings through errors inclusive. You can also use numeric forms like -p 3 for a single level.

Related Tutorials¶

• Linux – Category Overview
• systemd Service Management
• Disk and Filesystem Management (previous)
• Cron and Task Scheduling (next)
• Troubleshooting Linux Systems
• Learning Paths – DevOps Engineer

References¶

• systemd-journald documentation
• journalctl man page
• Linux man pages online – journalctl
• systemd journal configuration
• REBASH Academy – Linux Overview