Running Kubernetes in Production: Lessons Learned

15 min read
#kubernetes#infrastructure#devops#systems

Running Kubernetes in Production: Lessons Learned

Kubernetes is powerful but unforgiving. This post shares lessons from running production systems at scale: gotchas we hit, solutions we found, and practices that proved resilient.

Networking Gotchas

DNS Caching Hell

Kubernetes uses CoreDNS which caches DNS lookups. If a service moves to a new IP, clients may not see the change immediately.

The problem: A database pod restarts with a new IP. Your application's DNS resolver cached the old IP and keeps trying to connect to the dead pod for 30 minutes.

The solution:

# In your application deployment, reduce DNS TTL
dnsPolicy: ClusterFirst
dnsConfig:
  options:
  - name: ndots
    value: "2"
  - name: timeout
    value: "2"
  - name: attempts
    value: "3"

Add connection retry logic in your application. Don't rely on DNS alone.

Network Policy Gotchas

Network policies are stateless by default. Traffic from A to B doesn't automatically allow B to respond to A.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-db-traffic
spec:
  podSelector:
    matchLabels:
      app: database
  ingress:
  - from:
    - podSelector:
        matchLabels:
          tier: backend
    ports:
    - protocol: TCP
      port: 5432

This allows backend to reach database, but test both directions:

kubectl exec -it backend-pod -- psql -h database-service

CNI Plugin Pitfalls

Different CNI plugins have different performance and security characteristics. Flannel is simple but slow. Calico is faster but more complex.

Lesson learned: Test your CNI plugin under load before production. We switched from Flannel to Cilium and cut latency by 40%.

Storage and Persistence

PersistentVolume Binding Chaos

PVs can get stuck in Pending state if the storage class is misconfigured:

# Debug with
kubectl describe pvc my-claim
kubectl describe storageclass standard

Always define resource requests:

resources:
  requests:
    storage: 10Gi

Stateful Services Need Persistent Identity

StatefulSets are necessary for databases and queues. Regular Deployments don't guarantee stable hostnames:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless  # Required for stable DNS
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 100Gi

The headless service ensures each pod gets a stable DNS name like 

postgres-0.postgres-headless.default.svc.cluster.local
.

Security

RBAC Mistakes

Default RBAC is too permissive. A pod can potentially access the API server:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app

---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-app-role
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: my-app-role
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: default

Give each application only the permissions it needs. Start with read-only access and expand only if necessary.

Secret Management

Never commit secrets to git. Use a secret management system:

# Use external-secrets operator to fetch from Vault
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets-system

Define secrets sourced from external systems:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-password
    remoteRef:
      key: app/db-password

Observability

Missing Resource Requests

Without resource requests, Kubernetes can't schedule properly:

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

Requests are promises to Kubernetes. Set them based on actual usage, not guesses.

Logging Gotchas

Container logs disappear if a pod is evicted or deleted:

# Fetch logs from running pod
kubectl logs deployment/my-app

# Follow logs like tail -f
kubectl logs -f deployment/my-app

# Get logs from crashed container
kubectl logs my-pod --previous

Always use a centralized logging solution:

# Use Loki for simple logging
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack

Metrics Collection

Install Prometheus and scrape metrics from pods:

apiVersion: v1
kind: ServiceMonitor
metadata:
  name: app-metrics
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s

Expose a 

/metrics
endpoint in your application. Use a standard library:

import "github.com/prometheus/client_golang/prometheus"

requestsCounter := prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    },
    []string{"method", "endpoint"},
)

Deployment Patterns

Rolling Updates Require Health Checks

Without proper health checks, rolling updates can cause downtime:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
      - name: app
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Liveness checks whether to restart a pod. Readiness checks whether to receive traffic. Both are necessary.

Pod Disruption Budgets

Protect workloads from simultaneous evictions during maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

This ensures at least 2 pods remain available even during node drains.

Real Incident: Memory Leak Detection

We had a service leaking memory. Kubernetes would restart the pod periodically:

  1. Pod memory exceeded limit
  2. Kubernetes terminates pod (OOMKilled)
  3. Pod restarts

Detection:

kubectl top pod -A  # See memory usage

# Check restart count
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'

Fix:

  • Found unbounded cache in application
  • Added max size to cache
  • Added memory profiling in production
  • Problem solved with one code change

Conclusion

Kubernetes is a platform, not a set-and-forget solution. Key lessons:

  1. DNS can't be relied on alone. Add retry logic
  2. Define resource requests and limits based on real usage
  3. Use liveness and readiness probes
  4. Set up centralized logging and metrics immediately
  5. Test network policies and storage thoroughly
  6. RBAC should be as restrictive as possible
  7. Monitor for memory leaks and restart loops

These lessons come from real problems in production. They're not theoretical. Implement them early, and you'll avoid months of mysterious failures.

The best Kubernetes clusters aren't the most complex. They're the well-understood ones, with proper monitoring, clear deployment patterns, and a culture of observability.