When Distributed Systems Fail: Debugging in Production

Debugging distributed systems is fundamentally different from debugging single processes. You can't attach a debugger. You can't see internal state. All you have is logs, metrics, and traces. This post covers patterns that have solved real production failures.

The Three Pillars of Observability

1. Structured Logging

Log everything with context. Use structured fields, not free-form strings:

import "slog"

// Bad: unstructured
log.Printf("User %d logged in from %s", userID, ip)

// Good: structured
slog.Info("user login",
    slog.Int("user_id", userID),
    slog.String("ip", ip),
    slog.String("status", "success"),
)

In production, structured logs are queryable:

query: status=success and user_id=42
returns: [all login events for user 42]

2. Metrics

Metrics tell you when something is wrong. Logs tell you why.

Track these universally:

type MetricsCollector struct {
    requestCount    prometheus.Counter
    requestLatency  prometheus.Histogram
    activeRequests  prometheus.Gauge
    errorCount      prometheus.Counter
}

func (m *MetricsCollector) RecordRequest(duration time.Duration, err error) {
    m.requestCount.Inc()
    m.requestLatency.Observe(duration.Seconds())
    
    if err != nil {
        m.errorCount.Inc()
    }
}

Alert on these:

Error rate increasing
Latency P99 increasing
Error count spike

Case Study: Database Connection Exhaustion

Symptom: Requests suddenly start timing out.

Investigation:

Check active connections:

SELECT datname, count(*) 
FROM pg_stat_activity 
GROUP BY datname;

Check application logs:

# No errors initially. Then suddenly:
connection refused
too many open files

Check metrics:

# Sudden spike in active_db_connections from 15 to 95 (limit: 100)
# Latency percentile increases
# Error rate at 15%

Root cause: A goroutine was leaking database connections.

Debug approach:

// Add metric for connection pool size
activeConnections := prometheus.NewGaugeVec(
    prometheus.GaugeOpts{Name: "db_connections_active"},
    []string{"service"},
)

// Track by caller
// (Add to every sql.DB operation)
defer func() {
    stats := db.Stats()
    activeConnections.WithLabelValues("user_service").Set(float64(stats.OpenConnections))
}()

Distributed tracing revealed the culprit:

Service A -> (calls) -> Service B -> (db query) -> PostgreSQL
                            ^
                        Connection never returned to pool

A query in Service B was hanging indefinitely. Connection timed out but wasn't closed. Each new request opened a new connection. Eventually exhausted the pool.

Fix: Add context timeout:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

rows, err := db.QueryContext(ctx, "SELECT ...")

Case Study: Memory Leak Under Load

Symptom: Pod memory usage grows until OOMKilled. Restart loop.

Investigation:

Check memory metrics:

# Memory grows linearly over 4 hours
# No garbage collection dips
# Eventually hits limit (512MB)

Generate heap profile:

curl http://localhost:6060/debug/pprof/heap > heap.prof
go tool pprof heap.prof

Profile output:

(pprof) top
    1234.5MB of 1456.2MB total (84.77%)
        from 234 objects in main.cacheStore

Root cause: Cache had no eviction policy. Every cache.Set() added to memory indefinitely.

Fix:

type Cache struct {
    data map[string]interface{}
    size int64
    maxSize int64
}

func (c *Cache) Set(key string, value interface{}) {
    // Check size before adding
    if c.size > c.maxSize {
        c.evictOldest()
    }
    c.data[key] = value
    c.size += estimateSize(value)
}

Or use a battle-tested library:

import "github.com/patrickmn/go-cache"

cache := gcache.New(1000).LRU().Build()

Distributed Tracing Setup

OpenTelemetry gives you request flow across services:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger/jaegergrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

exporter, err := jaegergrpc.New(ctx,
    jaegergrpc.WithEndpoint("localhost:14250"),
)

provider := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
)
otel.SetTracerProvider(provider)

Instrument your handlers:

tracer := otel.Tracer("myapp")

func handleRequest(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "handleRequest")
    defer span.End()
    
    // Span is added to all child operations
    user := fetchUser(ctx, userID)
    orders := fetchOrders(ctx, userID)
    
    span.SetAttributes(
        attribute.Int("user_id", userID),
        attribute.Int("order_count", len(orders)),
    )
}

View traces in Jaeger UI. You'll see exactly which service is slow in a distributed call chain.

Log Aggregation Queries

When investigating, you'll write many log queries:

# Find all errors for service X in last hour
service="user-service" AND level="error" AND timestamp>now-1h

# Find correlation IDs matching a pattern
trace_id=abc123def456

# Find all timeout errors
error.kind="timeout"

# Find events for specific user
user_id=42 AND timestamp>now-1h

# Correlate multiple services
trace_id=xyz AND (service="api" OR service="db")

Most log systems (ELK, Loki, Datadog) support similar query syntax.

Building Resilience

Use these patterns to make systems debuggable:

Add Request IDs

type contextKey string
const requestIDKey contextKey = "request_id"

func extractRequestID(r *http.Request) string {
    if rid := r.Header.Get("X-Request-ID"); rid != "" {
        return rid
    }
    return uuid.New().String()
}

func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        rid := extractRequestID(r)
        ctx := context.WithValue(r.Context(), requestIDKey, rid)
        
        slog.Info("request start", slog.String("request_id", rid))
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

Include request ID in all logs. Now you can trace one request through all services.

Alert on Error Patterns

# Prometheus alert rule
- alert: HighErrorRate
  expr: rate(errors_total[5m]) > 0.05
  annotations:
    summary: "Error rate above 5%"

- alert: ConnectionPoolExhausted
  expr: db_connections_active > 80
  annotations:
    summary: "Database connection pool near limit"

Conclusion

Debugging distributed systems is investigative work. You form hypotheses based on metrics, test them with logs, and confirm with traces.

Tools you'll need:

Structured logging (ELK, Loki, Splunk)
Metrics (Prometheus)
Traces (Jaeger, Zipkin)
Profiling (pprof)
Alerting (Prometheus AlertManager)

Patterns that work:

Add request IDs to track requests across services
Use structured logging, not free-form strings
Instrument with metrics before failure happens
Implement distributed tracing early
Test failure scenarios in staging
Alert on leading indicators (latency, error rate)

The systems that recover fastest from failures aren't the most stable. They're the most observable.