Observability and Monitoring: Logs, Metrics, Traces, SLOs, and Alerts

Monitoring vs Observability

Monitoring tells you whether known things are healthy. Observability helps you understand unknown failure modes by inspecting system outputs: logs, metrics, traces, events, and profiles.

The Three Pillars

Modern observability also includes profiling, events, real-user monitoring, synthetic checks, and business metrics.

Structured Logging

Structured logs use fields instead of plain text.

{
  "level": "error",
  "service": "checkout",
  "traceId": "4bf92f",
  "userId": "u_123",
  "orderId": "ord_456",
  "message": "payment authorization failed",
  "provider": "stripe",
  "latencyMs": 382
}

Logging Rules

1. Include correlation IDs.
2. Log domain identifiers, not only stack traces.
3. Avoid secrets and sensitive data.
4. Use consistent fields across services.
5. Sample noisy success logs.
6. Keep error logs actionable.

Metrics

Metrics are numeric time-series used for dashboards and alerts.

RED Method

For request-driven services:

USE Method

For resources:

Use histograms for latency. Averages hide tail latency.

Distributed Tracing

Tracing follows one request across services.

OpenTelemetry is the common standard for instrumentation and exporting traces, metrics, and logs.

SLI, SLO, and SLA

Example:

SLI: percentage of checkout requests completed under 500 ms and without 5xx
SLO: 99.9% over 30 rolling days

Error Budget

If your SLO is 99.9%, your allowed failure budget is 0.1%. Error budgets help balance reliability work and feature velocity.

Alerting

Good alerts are actionable and tied to user impact.

Multi-Window Burn Rate

Burn-rate alerts catch fast incidents and slow reliability leaks.

Health Checks

Readiness should fail when critical dependencies are unavailable. Liveness should not fail just because a dependency is slow.

Dashboards

Create dashboards by user journey and service ownership.

Dashboards should answer operational questions quickly during incidents.

What to Remember for Interviews

1. Logs, metrics, and traces answer different questions.
2. Use RED for services and USE for resources.
3. Trace IDs connect distributed work: propagate them through every service call.
4. SLOs should reflect user experience: not just infrastructure health.
5. Alerts should be actionable: page on symptoms, investigate causes with dashboards.