Health Checks, Probes, and Self-Healing¶

Overview¶

Deploying a Pod is only the beginning. Production clusters must answer: Is this container alive? Is it ready to receive traffic? Has it finished starting up? Kubernetes answers these questions through probes — periodic health checks the kubelet runs against each container. Failed liveness probes trigger restarts; failed readiness probes remove the Pod from Service endpoints; startup probes protect slow-booting applications from premature liveness kills.

Combined with ReplicaSet controllers and node health monitoring, probes enable self-healing: crashed processes restart, unhealthy instances drain from load balancers, and the declared desired state converges without manual intervention. This is why Kubernetes replaces static "restart the service" runbooks with declarative health policies.

This is Tutorial 12 in Module 4: Networking & Operations of the REBASH Academy Kubernetes series. Complete Namespaces and Resource Management first. This tutorial follows our documentation standards.

Prerequisites¶

• Completed Namespaces and Resource Management
• Ability to deploy nginx or a simple HTTP app and expose it via a Service
• Understanding of Deployment rollouts and Pod lifecycle (Pending → Running → Terminating)
• Optional: curl inside the cluster or port-forward for probe verification
• Familiarity with YAML editing and kubectl apply

Learning Objectives¶

By the end of this tutorial, you will be able to:

• Distinguish liveness, readiness, and startup probes and when to use each
• Configure HTTP, TCP, and exec probe handlers in Pod specs
• Tune probe timing: initialDelaySeconds, periodSeconds, failureThreshold, timeoutSeconds
• Explain how readiness failures affect Service endpoints and Ingress routing
• Diagnose crash loops and probe-induced restart storms
• Design probe endpoints that reflect application health, not just process existence
• Relate probe behavior to Deployment rolling update success criteria

Architecture Diagram¶

The kubelet runs probes locally on each node. Readiness state flows to the endpoints controller; liveness failures trigger container restarts.

flowchart TB
    subgraph Node["Worker Node"]
        KL[kubelet]
        subgraph Pod["Pod: web"]
            C[Container nginx]
            LP[Liveness Probe]
            RP[Readiness Probe]
            SP[Startup Probe]
        end
        KL --> LP
        KL --> RP
        KL --> SP
        LP -->|HTTP GET /healthz| C
        RP -->|HTTP GET /ready| C
        SP -->|TCP :8080| C
    end

    API[API Server]
    EP[Endpoints Controller]
    SVC[Service web-svc]
    ING[Ingress]

    RP -->|Ready=true/false| API
    API --> EP
    EP --> SVC
    SVC --> ING
    LP -->|Fail threshold| RESTART[Container Restart]
    RESTART --> C

Theory¶

Why Probes Exist¶

Without probes, Kubernetes assumes a running process is healthy. A deadlocked Java application, a nginx master with zero workers, or a database still replaying WAL all appear "Running" to the container runtime. Probes give the control plane application-level signals:

Liveness Probes¶

Liveness detects dead containers. When failures exceed failureThreshold, the kubelet restarts the container. Use liveness for unrecoverable states — deadlock, corrupted in-memory state, hung event loops.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 2

Readiness Probes¶

Readiness determines traffic eligibility. A Pod can be Running but NotReady — for example, during warmup, cache loading, or leader election. The endpoints controller excludes NotReady pods from Service EndpointSlices; Ingress controllers stop routing to them.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

Readiness failures do not restart the container. The Pod stays Running until it passes again or is terminated by a rollout.

Startup Probes¶

Startup probes address slow-starting containers — large JVM heaps, database migrations, model loading. While startup is failing, liveness probes are disabled. Once startup succeeds, liveness takes over.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10
  # Allows up to 300 seconds to start

Without startup probes, you might set a large initialDelaySeconds on liveness — delaying crash detection for fast-fail scenarios. Startup probes decouple boot time from liveness sensitivity.

Probe Handler Types¶

HTTP probes support httpHeaders, scheme (HTTP/HTTPS), and success on status codes 200–399 by default.

Probe Timing Parameters¶

Total time before restart (liveness): roughly initialDelaySeconds + (periodSeconds × failureThreshold).

Self-Healing Mechanisms Beyond Probes¶

Probes are one layer of resilience:

flowchart LR
    RS[ReplicaSet] -->|desired replicas| POD[Pods]
    KL[kubelet] -->|liveness fail| RESTART[Restart container]
    NC[Node Controller] -->|node NotReady| RESCH[Reschedule pods]
    DC[Deployment Controller] -->|rollout| RS
    PDB[PodDisruptionBudget] -->|min available| RS

• ReplicaSet recreates deleted pods to match replicas
• Node controller evicts pods when nodes fail
• Deployment rolls out new versions with maxUnavailable / maxSurge controls
• PodDisruptionBudget limits voluntary disruptions during maintenance

Probes ensure individual instances are healthy; controllers maintain desired count and rollout safety.

Designing Health Endpoints¶

Production /healthz and /ready endpoints should differ:

Avoid expensive checks on liveness (full DB query across shards). Keep liveness fast and local; readiness can check dependencies with timeouts.

Hands-on Lab¶

Step 1 – Deploy nginx without probes (baseline)¶

Command:

kubectl create namespace probe-lab
kubectl create deployment web \
  --image=nginx:1.25-alpine \
  --replicas=2 \
  -n probe-lab
kubectl expose deployment web --port=80 -n probe-lab
kubectl get pods -n probe-lab -o wide
kubectl get endpoints web -n probe-lab

Explanation: Without readiness probes, pods join endpoints as soon as containers start — potentially before nginx binds port 80.

Expected output:

NAME                   READY   STATUS    RESTARTS   AGE
web-xxxxxxxxxx-xxxxx   1/1     Running   0          15s
web-xxxxxxxxxx-yyyyy   1/1     Running   0          15s

Step 2 – Add HTTP liveness and readiness probes¶

Command:

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: probe-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.25-alpine
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
            failureThreshold: 2
EOF

kubectl rollout status deployment/web -n probe-lab
kubectl describe pod -n probe-lab -l app=web | grep -A6 "Liveness\|Readiness"

Explanation: nginx serves / on port 80 — suitable for lab probes. Production apps expose dedicated /healthz endpoints.

Expected output:

Liveness:   http-get http://:80/ delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness:  http-get http://:80/ delay=0s timeout=1s period=5s #success=1 #failure=2

Step 3 – Observe readiness during rollout¶

Command:

kubectl set image deployment/web nginx=nginx:1.25-alpine -n probe-lab
kubectl get pods -n probe-lab -w &
sleep 2
kubectl get endpoints web -n probe-lab -o yaml | grep -A3 addresses
kill %1 2>/dev/null || true

Explanation: During rollout, new pods may show 0/1 Ready briefly. Endpoints update only for Ready pods — zero-downtime depends on readiness passing before old pods terminate.

Expected output:

web-aaa   0/1   Running   0   5s
web-aaa   1/1   Running   0   8s

Step 4 – Simulate liveness failure¶

Command:

POD=$(kubectl get pod -n probe-lab -l app=web -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n probe-lab "$POD" -- sh -c 'kill 1'
sleep 15
kubectl get pod -n probe-lab "$POD"
kubectl describe pod -n probe-lab "$POD" | tail -15

Explanation: Killing PID 1 in a container triggers liveness failure. The kubelet restarts the container — RESTARTS count increments.

Expected output:

NAME          READY   STATUS    RESTARTS      AGE
web-xxxxx     1/1     Running   1 (10s ago)   5m
Warning  Unhealthy  Liveness probe failed...
Normal   Killing    Container nginx failed liveness probe, will be restarted
Normal   Started    Started container nginx

Step 5 – Configure a startup probe for slow boot¶

Command:

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: slow-app
  namespace: probe-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: slow-app
  template:
    metadata:
      labels:
        app: slow-app
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c", "sleep 30 && while true; do echo ok | nc -l -p 8080; done"]
          ports:
            - containerPort: 8080
          startupProbe:
            tcpSocket:
              port: 8080
            periodSeconds: 5
            failureThreshold: 12
          livenessProbe:
            tcpSocket:
              port: 8080
            periodSeconds: 10
            failureThreshold: 3
EOF

kubectl get pods -n probe-lab -l app=slow-app -w &
sleep 35
kill %1 2>/dev/null || true
kubectl get pods -n probe-lab -l app=slow-app

Explanation: The app sleeps 30 seconds before listening. Startup probe allows 60 seconds (12 × 5s) without liveness killing the container during boot.

Expected output:

slow-app-xxxxx   0/1   Running   0   10s
slow-app-xxxxx   1/1   Running   0   35s

Step 6 – Use exec probe (optional pattern)¶

Command:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: exec-probe-demo
  namespace: probe-lab
spec:
  containers:
    - name: demo
      image: busybox:1.36
      command: ["sh", "-c", "touch /tmp/healthy && sleep 3600"]
      livenessProbe:
        exec:
          command:
            - cat
            - /tmp/healthy
        periodSeconds: 5
EOF

kubectl get pod exec-probe-demo -n probe-lab
kubectl exec -n probe-lab exec-probe-demo -- rm /tmp/healthy
sleep 20
kubectl get pod exec-probe-demo -n probe-lab

Explanation: Exec probes run commands inside the container. Removing /tmp/healthy simulates an unhealthy state — kubelet restarts the container.

Expected output:

exec-probe-demo   1/1   Running   1 (5s ago)   45s

Step 7 – Clean up¶

Command:

kubectl delete namespace probe-lab --wait=false

Commands & Code¶

Production probe template¶

containers:
  - name: api
    image: myorg/api:v2.4.1
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      failureThreshold: 30
      periodSeconds: 10
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      failureThreshold: 3
      timeoutSeconds: 2
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5
      failureThreshold: 2
      timeoutSeconds: 3

Common Mistakes¶

Best Practices¶

Troubleshooting¶

Summary¶

• Liveness probes restart unhealthy containers; readiness probes control Service traffic; startup probes protect slow boot sequences
• Probe handlers include HTTP, TCP, exec, and gRPC — choose based on application protocol
• Timing parameters (periodSeconds, failureThreshold, timeoutSeconds) determine sensitivity vs stability
• Self-healing combines probes with ReplicaSets, node monitoring, and Deployment rollouts
• Design separate health endpoints: liveness checks internal recovery; readiness checks request-serving ability
• Misconfigured probes cause restart storms — test under failure and load conditions before production

Interview Questions¶

1. What is the difference between a liveness probe and a readiness probe?
2. When would you use a startup probe instead of increasing initialDelaySeconds on liveness?
3. What happens to Service endpoints when a Pod fails its readiness probe?
4. Name three probe handler types and give a use case for each.
5. How does the kubelet act on consecutive liveness probe failures?
6. Why should liveness probes avoid checking external dependencies like databases?
7. Explain how readiness probes enable zero-downtime Deployments.
8. What is CrashLoopBackOff and how can probes contribute to it?
9. What HTTP status codes does Kubernetes consider successful for httpGet probes?
10. How do probes interact with PodDisruptionBudgets during node drains?

Related Tutorials¶

• Kubernetes – Category Overview
• Namespaces and Resource Management (previous — Module 4)
• RBAC and Kubernetes Security Basics (next — Module 5)
• Ingress and External Access
• Troubleshooting Kubernetes Workloads

References¶

• Configure Liveness, Readiness and Startup Probes
• Pod Lifecycle
• Pod Disruption Budgets
• Deployment – Rolling Update
• gRPC Health Checking Protocol
• REBASH Academy – Documentation Standards