Monitoring and Logging in Kubernetes¶

Overview¶

You cannot operate what you cannot see. Production Kubernetes requires metrics (is the cluster healthy?), logs (what happened when the pod crashed?), and traces (where did this request spend 2 seconds?). The CNCF stack — Prometheus, Grafana, Alertmanager, and Loki — is the de facto observability foundation, often deployed via kube-prometheus-stack Helm chart.

This tutorial covers the observability pillars for Kubernetes, ServiceMonitor patterns, log aggregation, golden signals, and alert design that pages humans only when action is required.

This is Tutorial 18 in Module 6: Production of the REBASH Academy Kubernetes series.

Prerequisites¶

• Production Patterns — HPA, PDB, and Affinity
• Health Checks, Probes, and Self-Healing
• Troubleshooting Kubernetes Workloads
• Helm Package Management
• Services and Cluster Networking
• Cluster with Helm 3 and sufficient resources (~4 GB RAM for monitoring stack)

Learning Objectives¶

By the end of this tutorial, you will be able to:

• Explain the three pillars of observability in a Kubernetes context
• Deploy kube-prometheus-stack and access Grafana dashboards
• Configure ServiceMonitors to scrape application metrics
• Aggregate container logs with Loki or the EFK pattern
• Define alerts on golden signals — latency, traffic, errors, saturation
• Correate metrics, logs, and Kubernetes events during incident response

Architecture Diagram¶

flowchart TB
    subgraph Apps["Application namespace"]
        API[votestack-api]
        WEB[votestack-web]
        API -->|/metrics| PROM
    end

    subgraph Observability["monitoring namespace"]
        PROM[Prometheus]
        AM[Alertmanager]
        GRAF[Grafana]
        LOKI[Loki]
        PROM --> AM
        PROM --> GRAF
        LOKI --> GRAF
    end

    subgraph Collection
        SM[ServiceMonitor]
        FB["Fluent Bit / Promtail"]
    end

    SM --> PROM
    API --> FB
    FB --> LOKI
    AM -->|PagerDuty / Slack| ONCALL[On-call engineer]

Theory¶

Three pillars of observability¶

Metrics are cheap at scale and drive alerts. Logs provide forensic detail. Traces connect microservice hops — essential for VoteStack's api → redis → worker → postgres flow.

Golden signals (Google SRE)¶

Add Availability (uptime) and Cost (resource usage) for platform dashboards.

Prometheus scraping model¶

Prometheus pulls metrics from targets on an interval. In Kubernetes:

1. Pods expose /metrics on a named port
2. ServiceMonitor CRD tells Prometheus Operator which services to scrape
3. PodMonitor scrapes pods directly (sidecar or hostNetwork patterns)

Key metric types:

Logging architecture¶

Containers log to stdout/stderr — Kubernetes captures via kubelet. Collection agents run as DaemonSet:

Structured JSON logs ({"level":"error","msg":"..."}) enable label-based filtering in Loki.

Alerting philosophy¶

Alert on symptoms, not causes. Page when users are impacted:

• ✅ Error rate > 1% for 5 minutes
• ✅ API p99 latency > 2s
• ❌ Single pod restart (use ticket, not page)
• ❌ Node NotReady for 30s (may self-heal)

Use Alertmanager for routing, inhibition, and silencing during maintenance.

Hands-on Lab¶

Lab 1 — Install kube-prometheus-stack¶

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --set grafana.adminPassword=changeme \
  --set prometheus.prometheusSpec.retention=15d \
  --wait

kubectl get pods -n monitoring
kubectl port-forward svc/kube-prometheus-grafana -n monitoring 3000:80
# Browser: http://localhost:3000 — admin / changeme

Explore built-in dashboards: Kubernetes / Compute Resources / Namespace, Node Exporter.

Lab 2 — Instrument VoteStack api¶

Add Prometheus client to the api (Node.js example):

const client = require('prom-client');
const register = new client.Registry();
client.collectDefaultMetrics({ register });

const httpRequests = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
  registers: [register],
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Rebuild and deploy via GitOps. Verify:

kubectl port-forward svc/votestack-api -n votestack 8080:8080
curl -s localhost:8080/metrics | head -20

Lab 3 — ServiceMonitor for api¶

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: votestack-api
  namespace: votestack
  labels:
    release: kube-prometheus   # must match Prometheus release label
spec:
  selector:
    matchLabels:
      app: votestack-api
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

Ensure the Service exposes port name http:

ports:
  - name: http
    port: 8080
    targetPort: 8080

Apply and confirm target in Prometheus UI → Status → Targets.

Lab 4 — Grafana dashboard panel¶

In Grafana, add panel:

• Query: rate(http_requests_total{namespace="votestack"}[5m])
• Legend: {{ method }} {{ route }}
• Add second panel: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Save dashboard to ConfigMap for GitOps:

apiVersion: v1
kind: ConfigMap
metadata:
  name: votestack-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  votestack.json: |
    { "... dashboard JSON ..." }

Lab 5 — Deploy Loki for log aggregation¶

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack -n monitoring \
  --set promtail.enabled=true \
  --set loki.persistence.enabled=true \
  --wait

Query logs in Grafana → Explore → Loki:

{namespace="votestack", app="votestack-api"} |= "error"
{namespace="votestack"} | json | level="error"

Correlate with metrics: click timestamp → view pod logs for same interval.

Lab 6 — Alert rules¶

PrometheusRule for high error rate:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: votestack-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
    - name: votestack
      rules:
        - alert: VoteStackApiHighErrorRate
          expr: |
            sum(rate(http_requests_total{namespace="votestack",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{namespace="votestack"}[5m]))
            > 0.01
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "VoteStack API error rate above 1%"
            runbook: "https://wiki.internal/runbooks/votestack-api-errors"
        - alert: VoteStackApiHighLatency
          expr: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{namespace="votestack"}[5m])) by (le)
            ) > 2
          for: 10m
          labels:
            severity: warning

Configure Alertmanager receiver (Slack webhook in Helm values):

alertmanager:
  config:
    receivers:
      - name: slack
        slack_configs:
          - channel: '#alerts'
            api_url: <webhook-url>
    route:
      receiver: slack
      routes:
        - match:
            severity: critical
          receiver: slack

Test: kubectl exec -n monitoring alertmanager-kube-prometheus-alertmanager-0 -- amtool alert add

Commands & Code¶

# Quick health checks
kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl get --raw /metrics | head

# Prometheus port-forward
kubectl port-forward svc/kube-prometheus-prometheus -n monitoring 9090:9090

# Recent events (correlate with metrics spikes)
kubectl get events -A --sort-by='.lastTimestamp' | tail -20

# Pod logs
kubectl logs -n votestack deploy/votestack-api --tail=100 -f
kubectl logs -n votestack <pod> --previous   # crashed container

# LogQL examples
# {namespace="votestack"} |= "panic"
# rate({namespace="votestack"}[1m])

Common Mistakes¶

Best Practices¶

Troubleshooting¶

Summary¶

• Metrics, logs, and traces form the observability pillars — Prometheus and Loki are the common Kubernetes stack
• Golden signals (latency, traffic, errors, saturation) drive meaningful alerts and dashboards
• ServiceMonitors declaratively configure Prometheus scraping for application /metrics endpoints
• Log aggregation via Promtail/Loki or Fluent Bit/EFK centralizes stdout logs from all pods
• Alert on symptoms with runbook links — reduce noise with for durations and severity routing
• Next: Kubernetes Security Hardening

Interview Questions¶

1. What are the three pillars of observability?
2. Explain the golden signals and give a Kubernetes example for each.
3. How does Prometheus discover scrape targets in Kubernetes?
4. What is a ServiceMonitor and why use it over static config?
5. Why should applications log to stdout instead of files?
6. How do you avoid alert fatigue in Kubernetes operations?
7. What is metric cardinality and why is it dangerous?
8. Compare PLG stack vs EFK stack for logging.
9. How would you debug a latency spike using metrics and logs together?
10. What is the difference between metrics-server and Prometheus?

Related Tutorials¶

• Production Patterns — HPA, PDB, and Affinity (previous)
• Kubernetes Security Hardening (next)
• Troubleshooting Kubernetes Workloads
• Container Logging and Monitoring
• Health Checks, Probes, and Self-Healing
• Kubernetes – Category Overview

References¶

• Prometheus – Documentation
• Grafana – Loki
• kube-prometheus-stack Helm chart
• Google SRE – Monitoring Distributed Systems
• OpenTelemetry – Kubernetes
• CNCF Observability Landscape