Remote systemd Service Control¶

Overview¶

When you operate a fleet of Linux servers, logging into each host to run systemctl restart nginx does not scale. systemd supports remote management over SSH via systemctl --host, and PolicyKit (polkit) lets you grant narrowly scoped privileges — for example, allowing a systemd-manager user to restart specific services without full root access.

This tutorial goes beyond surface-level examples: you will create a dedicated management user, configure SSH keys, write polkit rules that authorize only defined actions, and operate services remotely with the same commands you use locally.

This is Module 5, Tutorial 12 in the REBASH Academy Linux series.

Prerequisites¶

• SSH and Remote Administration — key-based auth and ~/.ssh/config
• systemd Service Management — systemctl, units, and journal basics
• Two Linux VMs (or one machine simulating remote via localhost and a second user) with systemd 230+
• sudo on the remote server to create users and polkit rules

Learning Objectives¶

By the end of this tutorial, you will be able to:

• Run systemctl and journalctl against remote hosts with --host user@server
• Create a least-privilege systemd-manager service account
• Write polkit rules granting specific systemd actions to non-root users
• Diagnose "Access denied" and SSH transport failures in remote systemctl
• Compare remote systemctl with configuration management and SSH ad-hoc patterns
• Apply security boundaries so operators cannot reload arbitrary units

Architecture Diagram¶

flowchart TD
    subgraph Admin["Workstation"]
        A[systemctl --host]
        B[journalctl --host]
    end
    subgraph SSH["Transport"]
        C[Encrypted SSH Session]
    end
    subgraph Remote["Server"]
        D[systemd-manager user]
        E["polkit / pkaction"]
        F[systemd PID 1]
        G[Unit Files]
        H[Running Services]
    end
    A --> C
    B --> C
    C --> D
    D --> E
    E -->|authorized| F
    E -->|denied| I[Access Denied]
    F --> G
    F --> H```

## Theory

### How `systemctl --host` works

When you run:

```bash
systemctl --host deploy@web01 status nginx

the local systemctl binary opens an SSH connection to web01 as user deploy, then executes the equivalent command in a pseudo-terminal or D-Bus proxy context on the remote side. The remote user must have permission to talk to systemd — typically via polkit rules or membership in a privileged group.

Requirements:

• SSH key auth working for user@host
• Remote machine running systemd as PID 1
• Compatible systemd versions (major mismatch can cause protocol errors)
• Polkit authorization for non-root users performing privileged actions

The same --host flag works with journalctl:

journalctl --host deploy@web01 -u nginx -n 20

Why not give everyone sudo?¶

sudo systemctl restart * is effectively root. Production teams want separation of duties:

Polkit expresses these as fine-grained rules evaluated at action time, logged in /var/log/auth.log or journal.

The systemd-manager user pattern¶

Create a dedicated account:

• No interactive login shell required (use /bin/bash or /sbin/nologin depending on policy)
• SSH key in ~/.ssh/authorized_keys — no password
• Member of groups only if needed; authorization primarily via polkit
• Home directory for SSH metadata only

This account is not a human user — it is a service identity for automation and jump-box workflows.

Polkit fundamentals¶

PolicyKit mediates privileged actions for unprivileged processes. systemd registers actions like:

• org.freedesktop.systemd1.manage-units — start, stop, restart units
• org.freedesktop.systemd1.reload-or-restart-unit
• org.freedesktop.systemd1.manage-unit-files — enable, disable (more sensitive)

Rules live in:

• /etc/polkit-1/rules.d/ — JavaScript-like .rules files (evaluated top-down, first match wins for allow/deny)
• /usr/share/polkit-1/actions/ — action definitions with defaults

Example rule structure:

polkit.addRule(function(action, subject) {
    if (action.id == "org.freedesktop.systemd1.manage-units" &&
        subject.user == "systemd-manager" &&
        action.lookup("unit") == "nginx.service") {
        return polkit.Result.YES;
    }
});

After adding rules: no daemon reload needed for polkit — rules are read on each action. Verify with pkaction and test remote commands.

Remote vs local systemctl differences¶

Unit file edits still happen on the server — remote systemctl does not sync files. Use Ansible, cloud-init, or CI/CD for unit file deployment; use remote systemctl for runtime control.

Security considerations¶

• Restrict systemd-manager SSH access by source IP in sshd_config (Match Address)
• Never share the management private key across people — use individual keys mapped to the same polkit-scoped user, or separate users per team
• Audit polkit and SSH logs centrally
• Deny daemon-reload and enable unless platform team requires it — those change boot behavior

Hands-on Lab¶

These steps assume a remote server (replace REMOTE with hostname/IP). For single-machine practice, use REMOTE=localhost and a second user on the same box.

export REMOTE=localhost   # change to your server IP/hostname
export MGMT_USER=systemd-manager

Step 1 – Create the systemd-manager user (on remote server)¶

Run on the remote server:

sudo useradd --create-home --comment "Remote systemd operator" "$MGMT_USER"
sudo mkdir -p /home/$MGMT_USER/.ssh
sudo chmod 700 /home/$MGMT_USER/.ssh
sudo chown -R $MGMT_USER:$MGMT_USER /home/$MGMT_USER/.ssh
id "$MGMT_USER"

Expected output: user uid/gid lines; home /home/systemd-manager.

Step 2 – Deploy SSH public key¶

From your workstation (key from SSH tutorial):

ssh-keygen -t ed25519 -f ~/.ssh/systemd_mgr_ed25519 -N "" -C "systemd-manager-key"
sudo bash -c "cat >> /home/$MGMT_USER/.ssh/authorized_keys" < ~/.ssh/systemd_mgr_ed25519.pub
sudo chmod 600 /home/$MGMT_USER/.ssh/authorized_keys
sudo chown $MGMT_USER:$MGMT_USER /home/$MGMT_USER/.ssh/authorized_keys

Test SSH:

ssh -i ~/.ssh/systemd_mgr_ed25519 ${MGMT_USER}@${REMOTE} 'whoami && hostname'

Expected output: systemd-manager and the server hostname.

Step 3 – Baseline remote status (expect permission errors)¶

Without polkit rules, privileged actions fail for non-root:

systemctl --host ${MGMT_USER}@${REMOTE} status ssh 2>/dev/null \
  || systemctl --host ${MGMT_USER}@${REMOTE} status sshd

Expected output: service status may work (read-only) on some distros, or polkit authentication prompt / access denied for restricted views.

Check failed units:

systemctl --host ${MGMT_USER}@${REMOTE} --failed --no-pager

Expected output: list of failed units or 0 loaded units listed.

Step 4 – Install polkit rule for nginx management¶

On the remote server, allow systemd-manager to manage only nginx.service (install nginx if needed for the lab):

sudo apt-get install -y nginx 2>/dev/null || sudo dnf install -y nginx 2>/dev/null || true

Create polkit rule:

sudo tee /etc/polkit-1/rules.d/50-systemd-manager-nginx.rules << 'EOF'
polkit.addRule(function(action, subject) {
    if (subject.user == "systemd-manager") {
        if (action.id == "org.freedesktop.systemd1.manage-units") {
            var unit = action.lookup("unit");
            if (unit == "nginx.service") {
                return polkit.Result.YES;
            }
        }
        if (action.id == "org.freedesktop.systemd1.manage-units" &&
            action.lookup("verb") == "reload-or-restart") {
            if (action.lookup("unit") == "nginx.service") {
                return polkit.Result.YES;
            }
        }
    }
});
EOF

List matching polkit actions:

pkaction | grep -i systemd | head -5

Expected output: action IDs like org.freedesktop.systemd1.manage-units.

Step 5 – Remote restart nginx¶

systemctl --host ${MGMT_USER}@${REMOTE} restart nginx
systemctl --host ${MGMT_USER}@${REMOTE} is-active nginx

Expected output: active

If nginx is not installed, substitute a running unit you scoped in polkit (e.g., a custom rebash-demo.service).

Step 6 – Verify authorization boundary¶

Attempt to restart sshd (not authorized):

systemctl --host ${MGMT_USER}@${REMOTE} restart sshd 2>&1 | head -3

Expected output: Access denied, Interactive authentication required, or polkit rejection — not a successful restart.

Step 7 – Remote journal access¶

journalctl --host ${MGMT_USER}@${REMOTE} -u nginx -n 5 --no-pager

Expected output: last five nginx log lines from the remote journal.

Follow logs remotely (Ctrl+C to stop):

timeout 3 journalctl --host ${MGMT_USER}@${REMOTE} -u nginx -f --no-pager || true

Expected output: streaming log lines or timeout exit after 3 seconds.

Step 8 – SSH config shortcut and fleet check script¶

Add to ~/.ssh/config:

Host web-fleet
    HostName YOUR_REMOTE_IP
    User systemd-manager
    IdentityFile ~/.ssh/systemd_mgr_ed25519

Fleet health script:

cat > ~/lab/check_fleet.sh << 'EOF'
#!/bin/bash
set -euo pipefail

HOSTS=(web-fleet)   # add more Host aliases

for h in "${HOSTS[@]}"; do
  printf '%-20s ' "$h"
  if systemctl --host "$h" is-system-running --wait 2>/dev/null; then
    failed=$(systemctl --host "$h" --failed --no-legend 2>/dev/null | wc -l)
    echo "running (failed units: $failed)"
  else
    echo "UNREACHABLE or degraded"
  fi
done
EOF
chmod +x ~/lab/check_fleet.sh

Run against configured host:

~/lab/check_fleet.sh

Expected output: host name with running and failed unit count.

Commands¶

Code¶

Polkit rule — read-only status for developers¶

// /etc/polkit-1/rules.d/40-dev-readonly.rules
polkit.addRule(function(action, subject) {
    if (subject.user == "dev-readonly" &&
        action.id == "org.freedesktop.systemd1.properties") {
        return polkit.Result.YES;
    }
});

Polkit rule — multiple allowed units¶

polkit.addRule(function(action, subject) {
    if (subject.user != "systemd-manager") return;

    var allowed = ["nginx.service", "app-backend.service", "worker.service"];
    if (action.id != "org.freedesktop.systemd1.manage-units") return;

    var unit = action.lookup("unit");
    if (allowed.indexOf(unit) >= 0) {
        return polkit.Result.YES;
    }
});

Ansible-style loop without Ansible¶

#!/bin/bash
set -euo pipefail

SERVICES=(nginx app-worker)
HOST=systemd-manager@prod-web01

for svc in "${SERVICES[@]}"; do
  echo "=== $svc ==="
  systemctl --host "$HOST" is-active "$svc"
done

Common Mistakes¶

Best Practices¶

Troubleshooting¶

Summary¶

• systemctl --host user@server and journalctl --host manage remote systemd over SSH — same CLI, different transport.
• A dedicated systemd-manager user with key-based SSH separates human admin accounts from automation identities.
• Polkit rules in /etc/polkit-1/rules.d/ grant least-privilege: specify user, action ID, and unit name — never blanket YES for manage-units.
• Unauthorized restart attempts should fail loudly; authorized restarts should succeed without sudo passwords.
• Remote control handles runtime; unit file deployment belongs in configuration management with reviewed changes.

Interview Questions¶

1. How does systemctl --host transport commands to a remote server?
2. Why create a dedicated systemd-manager user instead of sharing root SSH keys?
3. What is polkit, and how does it differ from sudo?
4. Name two polkit action IDs related to systemd unit management.
5. How would you allow a user to restart nginx.service but not sshd.service?
6. What happens if your polkit rule omits the .service suffix in the unit lookup?
7. How do you view remote logs for a unit without SSH-ing in manually?
8. Why should daemon-reload and enable be more restricted than restart?
9. What would you check first if systemctl --host user@host status nginx hangs?
10. Compare remote systemctl with running ssh user@host 'sudo systemctl restart nginx'.

Related Tutorials¶

• Linux – Category Overview
• SSH and Remote Administration (previous)
• Disk and Filesystem Management (next)
• systemd Service Management
• Learning Paths – DevOps Engineer

References¶

• systemd.systemctl man page — --host
• Polkit official documentation
• Red Hat — Managing systemd units with polkit
• systemd D-Bus API
• REBASH Academy – Linux Overview