Troubleshooting Kubernetes Workloads¶

Overview¶

When a Deployment fails at 2 AM, you need a systematic method — not random kubectl commands. Kubernetes exposes rich diagnostic signals: Pod status, Events, container logs, probe results, endpoint state, and node conditions. Advanced troubleshooting connects these layers: is the Pod scheduled? Is the container starting? Is the app healthy? Is the Service routing traffic? Is Ingress configured correctly?

This tutorial teaches the REBASH troubleshooting funnel — start broad (namespace events), narrow to the failing object (Pod → container), then verify dependencies (Service, DNS, RBAC). You will practice on deliberately broken workloads and build reflexes that SRE and platform engineers use daily.

This is Tutorial 14 in Module 5: Security & Tooling of the REBASH Academy Kubernetes series. Complete RBAC and Kubernetes Security Basics first. This tutorial follows our documentation standards.

Prerequisites¶

• Completed RBAC and Kubernetes Security Basics
• Familiarity with all Module 4 topics: Ingress, namespaces, quotas, and health probes
• Cluster access (minikube, kind, k3s, or cloud) where you can create failing workloads
• Comfort with kubectl describe, kubectl logs, and YAML editing
• Optional: Networking Troubleshooting Methodology for L3/L4 correlation

Learning Objectives¶

By the end of this tutorial, you will be able to:

• Apply a structured troubleshooting funnel from symptom to root cause
• Interpret Pod phases, conditions, and container waiting/terminated reasons
• Use Events and kubectl describe to diagnose scheduling and admission failures
• Debug CrashLoopBackOff, ImagePullBackOff, and probe-induced restart loops
• Verify Service → Endpoints → Pod connectivity with kubectl and in-cluster tests
• Use kubectl debug, ephemeral containers, and port-forward effectively
• Correlate Deployment rollout failures with ReplicaSet and revision history
• Document incidents with evidence suitable for postmortems

Architecture Diagram¶

Troubleshooting flows from the user-visible symptom down through Kubernetes layers to the container process.

flowchart TB
    SYM["Symptom<br/>502 / CrashLoop / Pending"]
    ING["Ingress / LoadBalancer"]
    SVC[Service]
    EP["Endpoints / EndpointSlice"]
    POD[Pod Status]
    CTR[Container State]
    APP[Application Logs]

    SYM --> ING
    ING -->|route OK?| SVC
    SVC -->|endpoints exist?| EP
    EP -->|targets ready?| POD
    POD -->|scheduled? probes?| CTR
    CTR -->|exit code / OOM| APP

    EVT[Events] -.-> POD
    EVT -.-> ING
    NODE[Node Conditions] -.-> POD

Theory¶

The Troubleshooting Funnel¶

Follow this order to avoid chasing red herrings:

Document findings at each step before diving deeper.

Pod Phases and Conditions¶

Conditions (Ready, PodScheduled, Initialized, ContainersReady) appear in kubectl describe pod. Ready=False with Running phase often indicates readiness probe failure — not a crash.

Container Waiting Reasons¶

Container Terminated Reasons¶

Check lastState.terminated.exitCode in Pod JSON for application-specific codes.

Events: The First Place to Look¶

Events are time-ordered, TTL-garbage-collected messages attached to objects:

kubectl get events -n <namespace> --sort-by='.lastTimestamp'
kubectl describe pod <name> -n <namespace>  # Events at bottom

Common event messages:

• FailedScheduling — insufficient CPU/memory, taints, affinity rules
• FailedMount — PVC not bound, wrong secret name
• Unhealthy — probe failures
• BackOff — restarting failed container with exponential delay

Scheduling Failures¶

When Pods stay Pending, the scheduler could not place them:

kubectl describe pod <pending-pod> | grep -A5 Events
kubectl describe nodes | grep -A10 "Allocated resources"
kubectl get pods -A --field-selector=status.phase=Pending

Causes: resource requests exceed node capacity, taints without tolerations, nodeSelector / affinity mismatches, ResourceQuota exhaustion, PVC unbound.

CrashLoopBackOff Investigation¶

kubectl logs <pod> -c <container> --previous
kubectl describe pod <pod> | grep -A3 "Last State"

Workflow:

1. Read --previous logs from last crashed instance
2. Check exit code (1 = app error, 137 = SIGKILL/OOM, 143 = SIGTERM)
3. Verify command, args, env vars, mounted Secrets
4. Temporarily disable liveness probe to distinguish crash vs probe kill
5. Run interactively: kubectl run debug --rm -it --image=<same-image> -- sh

Networking Troubleshooting¶

Traffic path: Ingress → Service → Endpoints → Pod IP:port

kubectl get ingress,svc,endpoints -n <ns>
kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- \
  wget -qO- http://<service>.<ns>.svc.cluster.local

Label selector mismatches between Service and Pod are a frequent subtle bug — app vs app.kubernetes.io/name.

Deployment Rollout Failures¶

kubectl rollout status deployment/<name> -n <ns>
kubectl rollout history deployment/<name> -n <ns>
kubectl get rs -n <ns> -l app=<label>

ProgressDeadlineExceeded means new ReplicaSet pods never became Ready within progressDeadlineSeconds. Compare old vs new Pod events and image tags.

Rollback: kubectl rollout undo deployment/<name> -n <ns>

Advanced Debug Techniques¶

For distroless images without shell, use debug containers or copy binary with kubectl debug node features.

RBAC and Admission Denials¶

Forbidden errors when applying manifests or from in-cluster clients:

kubectl auth can-i create deployments -n <ns> --as=<identity>
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Webhook timeouts manifest as intermittent apply failures — check API server logs on managed clusters via cloud console.

Hands-on Lab¶

Create a dedicated namespace and break things intentionally.

Step 1 – Set up troubleshooting namespace¶

Command:

kubectl create namespace troubleshoot-lab
kubectl config set-context --current --namespace=troubleshoot-lab
kubectl get events --sort-by='.lastTimestamp' | tail -5

Explanation: Start every incident by confirming namespace context and scanning recent events.

Expected output:

namespace/troubleshoot-lab created
LAST SEEN   TYPE     REASON    OBJECT   MESSAGE

Step 2 – Diagnose ImagePullBackOff¶

Command:

kubectl run bad-image --image=nginx:does-not-exist-tag
sleep 20
kubectl get pod bad-image
kubectl describe pod bad-image | tail -20
kubectl get events --field-selector involvedObject.name=bad-image

Explanation: Image pull failures show in container waiting.reason and Events with registry error details.

Expected output:

STATUS    REASON             MESSAGE
Waiting   ImagePullBackOff   Back-off pulling image "nginx:does-not-exist-tag"
Warning   Failed             Failed to pull image ...

Fix:

kubectl set image pod/bad-image bad-image=nginx:1.25-alpine
# Or delete and recreate — pod spec image is immutable on existing pod
kubectl delete pod bad-image --force --grace-period=0
kubectl run bad-image --image=nginx:1.25-alpine

Step 3 – Diagnose CrashLoopBackOff¶

Command:

kubectl run crasher --image=busybox:1.36 -- sh -c "echo starting && exit 1"
sleep 25
kubectl get pod crasher
kubectl logs crasher --previous 2>/dev/null || kubectl logs crasher
kubectl describe pod crasher | grep -A8 "Last State"

Explanation: Exit code 1 in logs confirms application failure. --previous captures the crashed instance during BackOff.

Expected output:

starting
State:     Waiting
  Reason:  CrashLoopBackOff
Last State: Terminated
  Reason:   Error
  Exit Code: 1

Step 4 – Diagnose Pending (resource pressure)¶

Command:

kubectl run hungry --image=nginx:1.25-alpine \
  --requests=cpu=999,memory=999Gi \
  --limits=cpu=999,memory=999Gi
sleep 10
kubectl get pod hungry
kubectl describe pod hungry | grep -A10 Events

Explanation: Unschedulable pods emit FailedScheduling events with exact resource shortfall.

Expected output:

Warning  FailedScheduling  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory

Fix:

kubectl delete pod hungry
kubectl run hungry --image=nginx:1.25-alpine --requests=cpu=50m,memory=64Mi

Step 5 – Diagnose Service selector mismatch¶

Command:

kubectl create deployment web --image=nginx:1.25-alpine --replicas=2
kubectl expose deployment web --port=80 --target-port=80
kubectl patch svc web -p '{"spec":{"selector":{"app":"wrong-label"}}}'
kubectl get endpoints web
kubectl run curl-test --image=curlimages/curl:8.5.0 --restart=Never -- \
  curl -s -o /dev/null -w "%{http_code}" --connect-timeout 3 http://web/ || true
sleep 5
kubectl logs curl-test 2>/dev/null; kubectl describe svc web | grep -A3 Selector

Explanation: Empty endpoints mean no Pod matches Service selector — traffic has nowhere to go.

Expected output:

Selector:  app=wrong-label
Endpoints: <none>
000  # curl connection failed

Fix:

kubectl patch svc web -p '{"spec":{"selector":{"app":"web"}}}'
kubectl get endpoints web

Step 6 – Diagnose readiness probe failure¶

Command:

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-fail
spec:
  replicas: 1
  selector:
    matchLabels:
      app: probe-fail
  template:
    metadata:
      labels:
        app: probe-fail
    spec:
      containers:
        - name: nginx
          image: nginx:1.25-alpine
          readinessProbe:
            httpGet:
              path: /nonexistent
              port: 80
            periodSeconds: 3
            failureThreshold: 1
EOF

sleep 15
kubectl get pods -l app=probe-fail
kubectl describe pod -l app=probe-fail | grep -A2 Unhealthy
kubectl get endpoints -l app=probe-fail 2>/dev/null || kubectl get endpoints

Explanation: Running but NotReady pods fail readiness — excluded from endpoints despite healthy process.

Expected output:

probe-fail-xxxxx   0/1   Running   0   15s
Warning  Unhealthy  Readiness probe failed: HTTP probe failed with statuscode: 404

Fix: Change probe path to / or implement /ready endpoint.

Step 7 – Practice rollback after bad rollout¶

Command:

kubectl set image deployment/web web=nginx:1.25-alpine
kubectl rollout status deployment/web
kubectl set image deployment/web web=nginx:bad-tag-xyz
sleep 30
kubectl rollout status deployment/web --timeout=20s || true
kubectl get rs -l app=web
kubectl rollout undo deployment/web
kubectl rollout status deployment/web

Explanation: Failed rollouts leave old ReplicaSet scaled down but available. rollout undo reverts to previous revision.

Expected output:

error: deployment "web" exceeded its progress deadline
deployment.apps/web rolled back
deployment "web" successfully rolled out

Step 8 – Clean up and document¶

Command:

kubectl delete namespace troubleshoot-lab --wait=false
kubectl config set-context --current --namespace=default
echo "Incident template: Symptom → Scope → Pod → Container → Network → Root cause → Fix"

Commands & Code¶

Troubleshooting checklist script¶

Save as ~/bin/k8s-triage.sh:

#!/usr/bin/env bash
# k8s-triage.sh — quick workload triage
set -euo pipefail
NS="${1:-default}"
APP="${2:-}"

section() { printf '\n=== %s ===\n' "$1"; }

section "Context"
kubectl config current-context
kubectl config view --minify -o jsonpath='{.contexts[0].context.namespace}{"\n"}' 2>/dev/null || echo "default"

section "Recent Events ($NS)"
kubectl get events -n "$NS" --sort-by='.lastTimestamp' | tail -15

section "Problem Pods ($NS)"
kubectl get pods -n "$NS" --field-selector=status.phase!=Running,status.phase!=Succeeded 2>/dev/null || \
  kubectl get pods -n "$NS" | grep -vE 'Running|Completed' || echo "None"

if [ -n "$APP" ]; then
  section "Deployment ($APP)"
  kubectl get deploy,rs,pods,svc,endpoints -n "$NS" -l "app=$APP" 2>/dev/null || true
fi

Usage: chmod +x ~/bin/k8s-triage.sh && ~/bin/k8s-triage.sh troubleshoot-lab web

Common Mistakes¶

Best Practices¶

Troubleshooting¶

Summary¶

• Use a structured funnel: scope → workload → pod → container → network → cluster
• Events and kubectl describe reveal scheduling, mount, and probe failures before logs help
• CrashLoopBackOff requires --previous logs and exit code analysis; distinguish OOM from app errors
• Empty endpoints usually mean label selector mismatch or readiness failure — not Ingress issues
• Rollout undo restores service; always capture failing revision for root cause analysis
• Advanced tools (kubectl debug, port-forward, triage scripts) accelerate diagnosis under pressure

Interview Questions¶

1. What is the first command or information source you check when a Deployment is failing?
2. Explain the difference between Pod phase Running and condition Ready=False.
3. How do you retrieve logs from a container that crashed and restarted?
4. What causes ImagePullBackOff and how do you diagnose it?
5. A Service exists but has no endpoints. List three possible causes.
6. What does exit code 137 typically indicate in a terminated container?
7. Describe the traffic path from Ingress to Pod and where to inspect each hop.
8. What is ProgressDeadlineExceeded on a Deployment?
9. When would you use kubectl debug instead of kubectl exec?
10. How do ResourceQuota and scheduling failures appear in Pod Events?

Related Tutorials¶

• Kubernetes – Category Overview
• RBAC and Kubernetes Security Basics (previous — Module 5)
• Helm Package Management (next in Module 5)
• Health Checks, Probes, and Self-Healing
• Networking – Network Troubleshooting Methodology

References¶

• Debug Running Pods
• Debug Init Containers
• Determine Pod Failure Reason
• Troubleshoot Applications
• kubectl Reference – logs, describe, events
• Deployment – Rolling Back
• REBASH Academy – Documentation Standards