Basic System Concepts: Latency, Throughput, Scaling, Load Balancing, and Caching

The Blueprint of System Design

In system design, every decision is a trade-off. To build systems that scale to millions of users, you must understand the fundamental forces that govern distributed architecture. These aren't just academic concepts—they are the "blueprint" used by companies like Google, Meta, and Amazon to build global-scale infrastructure.

---

1. The Performance Dial: Latency vs. Throughput

Engineers often use "speed" as a catch-all term, but in distributed systems, speed has two distinct dimensions:

• Latency: The time it takes for a single request to complete (measured in ms, µs, ns). It's the "delay."
• Throughput: The number of requests a system can handle per unit of time (measured in req/s, QPS, or MB/s). It's the "capacity."

Latency Numbers Every Programmer Should Know

To build intuition, you must understand the relative cost of operations. Modern computers are fast, but the gap between "local" and "network" is monumental.

---

2. The Scaling Spectrum: Vertical vs. Horizontal

When your system hits its limits, you have two primary ways to grow.

Vertical Scaling (Scaling Up)

Adding more power (CPU, RAM) to an existing machine.

• Pros: simple, no code changes, low complexity.
• Cons: upper hardware limit, single point of failure (SPOF), expensive.
• Real World: Instagram famously ran on a single massive PostgreSQL instance for years before sharding.

Horizontal Scaling (Scaling Out)

Adding more machines to the pool.

• Pros: No upper limit, resilient (no SPOF), uses commodity hardware.
• Cons: Requires a Load Balancer, introduces distributed complexity (consistency, network partitions).
• Real World: Airbnb improved the scalability of its web-serving tier by removing bottlenecks and distributing traffic across

---

3. Reliability & Availability: "The Nines"

A system can be fast and scalable, but it's useless if it's down.

• Reliability: The probability that a system will perform its intended function without failure for a specified period.
• Availability: The percentage of time a system is operational and accessible.

Availability is measured in "Nines":

---

4. Traffic Distribution: Load Balancing

Once you scale horizontally, you need a Load Balancer (LB) to act as a "traffic cop."

Layer 4 vs. Layer 7

• L4 (Transport): Routes based on IP and Port. Extremely fast, but "blind" to the application data.
• L7 (Application): Routes based on URLs, Cookies, and Headers. "Smart" but slower.
  • Example: Send all /images/* requests to a specialized image server, and /api/* to the backend.

Consistent Hashing

In a distributed system, how do you decide which server gets which piece of data? Simple modulo (key % n) fails when you add or remove servers (it reshuffles everything).

Consistent Hashing maps keys and servers to a logical "ring," ensuring that adding/removing a node only affects 1/n of the keys.

• Classic Paper: Amazon's Dynamo Paper.
• Case Study: Discord uses this to scale their real-time communication.

---

5. The Golden Layer: Caching

Caching is the single most effective way to improve performance. By storing frequently accessed data in memory (RAM), you avoid expensive database or disk operations.

Where to Cache?

1. Client Side: Browser cache (HTTP headers).
2. CDN (Edge): Static assets (images, JS) cached closer to users via providers like Cloudflare or Fastly.
3. Application Layer: In-memory stores like Redis or Memcached.

The Cache-Aside Pattern

The most common strategy for general-purpose applications:

1. Read: Check Cache. If MISS, read from DB, write to Cache, and return.
2. Write: Update DB first, then invalidate (delete) the cache entry.

Scaling Success Stories

• Slack: Scaled their MySQL fleet using Vitess for horizontal sharding.
• Google: Manages millions of containers using Borg, the predecessor to Kubernetes.

---

Summary: The Interview Cheat Sheet