Reliability Patterns: Circuit Breakers, Retries, Fallbacks, and Graceful Degradation

Why Reliability Matters

Every system fails eventually. The question is whether your system degrades gracefully or collapses completely. Reliability patterns help you build systems that survive failures.

Failure Modes

Types of Failures

Timeouts

Always set timeouts. Without them, a slow service can cause your entire system to hang.

Timeout Strategy

Setting Timeouts

The code configures HTTP request timeouts using httpx with a 5-second total timeout and 1-second connection timeout. Always setting timeouts prevents requests from waiting indefinitely when services fail.

Retry Patterns

When to Retry

Exponential Backoff

Exponential backoff increases wait time between retries exponentially (1s, 2s, 4s...) with random jitter to prevent thundering herd. The jitter adds randomness to avoid synchronized retries from multiple clients.

Circuit Breaker Pattern

Prevent cascading failures by stopping calls to a failing service.

States

Implementation

The circuit breaker tracks failures and opens the circuit when a threshold is reached. In the closed state (normal), it executes the function. In the open state, it rejects calls immediately. After a timeout, it moves to half-open state to test if the service has recovered.

Circuit Breaker in Action

Bulkhead Pattern

Isolate components so that failure in one doesn't affect others.

Fallback and Graceful Degradation

Fallback Strategies

Code Example

def get_product_recommendations(user_id):
    try:
        # Try primary service
        recommendations = ml_service.get_recommendations(user_id)
        return recommendations
    except MLServiceError:
        # Fallback 1: Cached recommendations
        cached = cache.get(f"recs:{user_id}")
        if cached:
            return cached
        
        # Fallback 2: Popular products
        return popular_products()

Graceful Degradation Matrix

Health Checks

Helpful for orchestration systems to know when to route traffic or restart services.

Health Check Levels

Health Check Implementation

from fastapi import FastAPI

app = FastAPI()

@app.get("/health")
async def health():
    checks = {
        "database": check_database(),
        "cache": check_cache(),
        "external_api": check_external_api(),
    }
    
    all_healthy = all(checks.values())
    
    return {
        "status": "healthy" if all_healthy else "unhealthy",
        "checks": checks
    }, 200 if all_healthy else 503

Chaos Engineering

Test your reliability by deliberately breaking things.

Reliability Checklist

What to Remember for Interviews

1. Timeouts are essential: Always set them; infinite waits are never acceptable
2. Retry with backoff: Exponential backoff + jitter prevents thundering herd
3. Circuit breakers: Protect against cascading failures
4. Graceful degradation: Have fallback strategies for every critical path
5. Test for failure: Use chaos engineering to verify reliability