Observability Basics: Logs, Metrics, and Traces

Observability isn't about collecting more data—it's about asking questions you didn't know you'd need to ask. Here's how to build a foundation that actually helps you debug production issues.

The Three Pillars

Logs: What Happened

Logs are discrete events with context. They tell you what happened at a specific point in time.

{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123",
  "user_id": "user_456",
  "message": "Payment processing failed",
  "error": "Card declined",
  "amount": 99.99,
  "currency": "USD"
}

Best Practices:

Use structured logging (JSON, not plain text)
Include correlation IDs (trace_id, request_id)
Add business context (user_id, order_id)
Use appropriate log levels

Metrics: How Much

Metrics are numerical measurements aggregated over time. They tell you how your system is performing.

# Counter: Total requests
http_requests_total{method="POST", path="/api/orders", status="200"} 15420

# Gauge: Current connections
database_connections_active{pool="primary"} 23

# Histogram: Request latency distribution
http_request_duration_seconds_bucket{le="0.1"} 8000
http_request_duration_seconds_bucket{le="0.5"} 9500
http_request_duration_seconds_bucket{le="1.0"} 9800

Key Metrics to Track:

Request rate (requests/second)
Error rate (errors/second or %)
Latency (p50, p95, p99)
Saturation (CPU, memory, connections)

Traces: Why It's Slow

Traces follow a request through your distributed system, showing you where time is spent.

Trace ID: abc123
├── frontend-web (32ms)
│   └── api-gateway (28ms)
│       ├── auth-service (5ms)
│       └── order-service (20ms)
│           ├── inventory-check (8ms)
│           └── database-query (10ms)

Implementation Strategy

Step 1: Structured Logging

Transform unstructured logs to structured:

# Before: Hard to parse, no context
logger.info(f"User {user_id} placed order {order_id}")

# After: Structured with context
logger.info(
    "Order placed",
    extra={
        "user_id": user_id,
        "order_id": order_id,
        "amount": amount,
        "trace_id": get_trace_id()
    }
)

Step 2: RED Metrics

Implement the RED method for every service:

Rate: Requests per second
Errors: Failed requests per second
Duration: Latency distribution

from prometheus_client import Counter, Histogram

REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency',
    ['method', 'endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
)

Step 3: Distributed Tracing

Propagate trace context across services:

from opentelemetry import trace
from opentelemetry.propagate import inject, extract

tracer = trace.get_tracer(__name__)

# Start a span
with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.amount", amount)

    # Propagate context to downstream services
    headers = {}
    inject(headers)
    response = requests.post(
        "http://inventory/check",
        headers=headers
    )

Correlation: Connecting the Dots

The real power comes from connecting logs, metrics, and traces:

{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "level": "error",
  "trace_id": "abc123",        // Links to trace
  "span_id": "span456",        // Specific operation
  "service": "payment-api",
  "message": "Payment failed"
}

With this trace_id, you can:

Find all logs from this request
See the full trace in Jaeger/Tempo
Correlate with metrics for that time window

Tool Stack Recommendations

Budget-Friendly (Self-Hosted)

Pillar	Tool	Notes
Logs	Loki	Pairs well with Grafana
Metrics	Prometheus	Industry standard
Traces	Jaeger or Tempo	Tempo if using Loki
Visualization	Grafana	Unified dashboards

Managed Services

Pillar	AWS	GCP	Agnostic
Logs	CloudWatch	Cloud Logging	Datadog
Metrics	CloudWatch	Cloud Monitoring	Datadog
Traces	X-Ray	Cloud Trace	Datadog

Common Mistakes

1. Logging Everything

More logs != better observability. Log meaningful events, not every function call.

# Bad: Noise
logger.debug("Entering function")
logger.debug("Variable x = 5")
logger.debug("Exiting function")

# Good: Meaningful events
logger.info("Payment processed", extra={"amount": 100, "user_id": "123"})

2. Ignoring Cardinality

High-cardinality labels explode metric storage:

# Bad: user_id has millions of unique values
REQUEST_COUNT = Counter(
    'requests_total',
    ['method', 'endpoint', 'user_id']  # Don't do this
)

# Good: Keep cardinality bounded
REQUEST_COUNT = Counter(
    'requests_total',
    ['method', 'endpoint', 'status']
)

3. No Sampling for Traces

Tracing 100% of requests is expensive. Sample intelligently:

# Sample 10% of requests, but always trace errors
sampler = TraceIdRatioBased(0.1)  # 10% sampling

Alert on Symptoms, Not Causes

Good alerts focus on user impact:

# Good: Alerts on user-facing symptoms
- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total[5m])) > 0.01
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Error rate above 1%"

# Bad: Alerts on implementation details
- alert: DatabaseConnectionsHigh
  expr: database_connections > 80

Quick Start Checklist

Implement structured JSON logging
Add trace_id to all log entries
Expose RED metrics for every service
Set up distributed tracing with context propagation
Create a unified dashboard in Grafana
Configure alerts for error rate and latency
Document runbooks for common alerts

Key Takeaways

Structured logs with correlation IDs are non-negotiable
RED metrics give you the health of every service at a glance
Traces show you where latency hides in distributed systems
Correlation between all three is where the magic happens
Alert on symptoms, investigate with observability data

Observability isn't a product you buy—it's a practice you build. Start with the basics, iterate, and expand as your understanding of your system grows.