Kubernetes Security Hardening¶

Overview¶

Kubernetes defaults prioritize developer velocity over maximum security. Production clusters require deliberate hardening: Pod Security Standards restrict dangerous pod capabilities, NetworkPolicies enforce micro-segmentation, admission controllers block non-compliant resources at the API gate, and RBAC grants least-privilege access. Supply-chain controls ensure only trusted, scanned images run in the cluster.

This tutorial builds a defense-in-depth security model for the VoteStack application and cluster infrastructure — the capstone security layer before full production deployment.

This is Tutorial 19 in Module 6: Production of the REBASH Academy Kubernetes series.

Prerequisites¶

• Monitoring and Logging in Kubernetes
• RBAC and Kubernetes Security Basics
• ConfigMaps and Secrets
• Namespaces and Resource Management
• GitOps and CI/CD with Kubernetes
• Cluster admin access for policy installation
• Optional: Docker Security Hardening

Learning Objectives¶

By the end of this tutorial, you will be able to:

• Apply Pod Security Standards (restricted, baseline, privileged) at namespace level
• Write NetworkPolicies that default-deny and allow explicit traffic paths
• Configure admission policies with Kyverno or OPA Gatekeeper
• Implement RBAC least privilege for users, CI, and GitOps agents
• Secure secrets with External Secrets Operator or Sealed Secrets
• Enforce image provenance and vulnerability thresholds in the deploy pipeline

Architecture Diagram¶

flowchart TB
    subgraph Perimeter
        ING[Ingress + TLS]
        WAF["WAF / rate limit"]
    end

    subgraph Admission
        API[Kubernetes API]
        KYV["Kyverno / Gatekeeper"]
        API --> KYV
    end

    subgraph Namespace["votestack namespace"]
        NETP[NetworkPolicy]
        PSS["Pod Security: restricted"]
        WEB[web]
        APIP[api]
        WORK[worker]
        WEB --> NETP
        APIP --> NETP
        WORK --> NETP
    end

    subgraph Data
        PG["(postgres")]
        RD["(redis")]
    end

    ING --> WEB
    ING --> APIP
    APIP --> RD
    APIP --> PG
    WORK --> RD
    WORK --> PG
    KYV --> Namespace

Theory¶

Defense in depth for Kubernetes¶

No single layer is sufficient — assume breach and limit blast radius.

Pod Security Standards¶

Replace deprecated PodSecurityPolicy with Pod Security Admission (built-in):

Apply via namespace labels:

metadata:
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Pods violating enforce are rejected; warn surfaces issues in audit logs.

NetworkPolicy fundamentals¶

Without policies, all pods communicate freely (default allow). Production uses default deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Then add explicit allow rules — ingress controller → web/api; api → postgres/redis only.

Requires a CNI that enforces NetworkPolicy (Calico, Cilium, Weave).

Admission control¶

Validating policies reject bad resources. Mutating policies inject defaults (labels, sidecars).

Kyverno example — require resource limits:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "CPU and memory requests/limits required"
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    memory: "?*"
                    cpu: "?*"
                  limits:
                    memory: "?*"
                    cpu: "?*"

RBAC least privilege¶

Avoid cluster-admin bindings except for bootstrap automation with audited access.

Secrets management¶

Never commit plaintext Secrets to Git. Options:

Enable encryption at rest for etcd Secret resources via KMS provider.

Hands-on Lab¶

Lab 1 — Apply Pod Security Standards¶

Label the votestack namespace:

kubectl label namespace votestack \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted \
  --overwrite

Update VoteStack api Deployment for restricted compliance:

spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

Apply and verify: kubectl apply -f api-deployment.yaml

Test rejection — this pod should fail:

# bad-pod.yaml — privileged
securityContext:
  privileged: true

kubectl apply -f bad-pod.yaml -n votestack
# Expected: forbidden by PodSecurity

Lab 2 — Default-deny NetworkPolicies¶

# 01-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: votestack
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# 02-allow-api-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-ingress
  namespace: votestack
spec:
  podSelector:
    matchLabels:
      app: votestack-api
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
        - podSelector:
            matchLabels:
              app: votestack-web
      ports:
        - protocol: TCP
          port: 8080
---
# 03-allow-api-egress-data.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-egress-data
  namespace: votestack
spec:
  podSelector:
    matchLabels:
      app: votestack-api
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    - to:   # DNS
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

kubectl apply -f networkpolicies/
kubectl run -it debug --rm --image=busybox -n votestack -- wget -qO- http://votestack-api:8080/health
# Should succeed from allowed paths; fail from unauthorized pods

Lab 3 — Install Kyverno and enforce policies¶

helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno -n kyverno --create-namespace --wait
kubectl get clusterpolicies

Apply policies from Theory section plus disallow :latest tag:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "Using ':latest' tag is not allowed"
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Test: deploy pod with nginx:latest — should be blocked.

Lab 4 — Sealed Secrets for database credentials¶

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system --wait

kubectl create secret generic postgres-credentials \
  --from-literal=username=voteapp \
  --from-literal=password='super-secret' \
  -n votestack --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-postgres.yaml

kubectl apply -f sealed-postgres.yaml
# Safe to commit sealed-postgres.yaml to GitOps repo

Reference in Deployment:

envFrom:
  - secretRef:
      name: postgres-credentials

Lab 5 — RBAC for GitOps agent¶

Argo CD should not use cluster-admin. Create scoped Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd-votestack
  namespace: votestack
rules:
  - apiGroups: ["", "apps", "networking.k8s.io"]
    resources: ["*"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd-votestack
  namespace: votestack
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd-votestack
subjects:
  - kind: ServiceAccount
    name: argocd-application-controller
    namespace: argocd

Lab 6 — Image scanning in CI + admission¶

CI gate (GitHub Actions):

- name: Trivy scan
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: ghcr.io/org/votestack-api:${{ github.sha }}
    severity: CRITICAL,HIGH
    exit-code: 1

Kyverno verifyImages (with cosign signatures in mature setups):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "ghcr.io/org/votestack-*"
          attestors:
            - count: 1
              entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      ...
                      -----END PUBLIC KEY-----

Commands & Code¶

# Pod Security
kubectl label namespace votestack pod-security.kubernetes.io/enforce=restricted
kubectl auth can-i create pods --as=system:serviceaccount:votestack:default -n votestack

# NetworkPolicy debug
kubectl run nettest --rm -it --image=nicolaka/netshoot -n votestack
# curl, nc, traceroute between services

# RBAC audit
kubectl auth can-i --list --as=developer@corp.com -n votestack
kubectl get rolebindings,clusterrolebindings -A | grep argocd

# Audit logs (enable on api-server)
# --audit-log-path=/var/log/k8s-audit.log

# Runtime security (optional)
helm install falco falcosecurity/falco -n falco --create-namespace

Common Mistakes¶

Best Practices¶

Troubleshooting¶

Summary¶

• Defense in depth combines RBAC, admission control, pod hardening, network segmentation, and supply-chain verification
• Pod Security Standards enforce restricted profiles in application namespaces
• NetworkPolicies implement default-deny with explicit allow paths for VoteStack tiers
• Kyverno/Gatekeeper block non-compliant manifests before pods are scheduled
• Sealed Secrets or External Secrets keep credentials out of plaintext Git
• Next: Kubernetes Capstone and Next Steps

Interview Questions¶

1. What are the three Pod Security Standard profiles and when use each?
2. Why implement default-deny NetworkPolicies?
3. How does admission control differ from RBAC?
4. Explain the blast radius if Argo CD has cluster-admin and the GitOps repo is compromised.
5. What is seccomp RuntimeDefault and why enable it?
6. Compare Sealed Secrets vs External Secrets Operator.
7. How do you allow DNS in a default-deny egress NetworkPolicy?
8. What supply-chain controls apply before an image runs in Kubernetes?
9. How would you audit who deleted a production Deployment?
10. What is the difference between Pod Security Admission and Kyverno?

Related Tutorials¶

• Monitoring and Logging in Kubernetes (previous)
• Kubernetes Capstone and Next Steps (next)
• RBAC and Kubernetes Security Basics
• ConfigMaps and Secrets
• Docker Security Hardening
• Network Security Hardening
• Kubernetes – Category Overview

References¶

• Kubernetes – Pod Security Standards
• Kubernetes – Network Policies
• Kyverno – Policies
• OPA Gatekeeper
• Sealed Secrets
• CIS Kubernetes Benchmark
• NSA/CISA Kubernetes Hardening Guide