System Design from first principles

System design isn't a list of buzzwords to memorize — it's a small set of heuristics that keep repeating: spread load, tolerate failure, pick your trade-off consciously, make retries safe. Once you see the pattern, every "new" system is a remix of the same fifteen ideas.

Each chapter below is one mechanism, with one thing you can click and break. Do the breaking — that's where the lesson lives, not in the paragraph above it.

Vertical scaling means a bigger machine — more CPU, more RAM. It's simple and it works, but it has a ceiling (the biggest machine money can buy) and a single point of failure. Horizontal scaling means more machines doing the same job behind a load balancer. It has (almost) no ceiling, but it only works if any request can be handled by any machine — which is why later chapters spend so much effort making services stateless.

Before scaling anything, estimate the load: requests per second, data per day, read/write ratio. A back-of-envelope number tells you whether you need three servers or three thousand — and whether the bottleneck is CPU, network, or disk.

Reliability is what happens after something breaks — and something always breaks. The fix is redundancy: no component that matters should exist as a single copy. A single point of failure (SPOF) is any one thing whose death takes the whole system down with it.

Availability is usually expressed in "nines": 99.9% ("three nines") allows about 8.8 hours of downtime a year; 99.99% allows about 52 minutes. Each extra nine costs real engineering effort, so pick a target the business actually needs — not the highest number that sounds impressive.

The CAP theorem says: when a network Partition happens between nodes that hold the same data, you must choose between Consistency (every read sees the latest write, so you refuse writes on the cut-off side) and Availability (every request gets a response, even if it might be stale). Outside of a partition, you can have both — CAP is specifically about what happens during a network split, not a permanent trade-off.

A related, more practical framing is PACELC: if there is a Partition, choose Availability or Consistency; Else (normal operation), choose Latency or Consistency. Strongly consistent systems pay a latency tax even when nothing is broken, because they must coordinate before answering.

A load balancer sits in front of a pool of identical servers and picks one for each request. Round robin cycles through servers in order — simple, ignores current load. Least connections sends the request to whichever server has the fewest in-flight requests — better when requests take unequal time. Consistent hashing maps both servers and requests onto a ring by hash, so a given key (e.g. a user ID) usually lands on the same server — useful for cache locality — and when a server dies, only its slice of the ring remaps, not everything.

Load balancers also work at different layers: L4 balances raw TCP/UDP connections (fast, protocol-blind); L7 reads HTTP (host, path, headers) and can route /api/* to one pool and /static/* to another. Either way, a load balancer must stop sending traffic to a node that fails its health check.

A cache trades a little staleness for a lot of speed. Cache-aside (lazy loading): the app checks the cache first; on a miss, it reads the database and writes the result into the cache. Write-through: every write goes to the cache and the database together, so the cache is never stale but writes are slower. Write-behind: writes land in the cache first and are flushed to the database asynchronously — fast, but can lose data if the cache dies before flushing.

Every cache entry needs an eviction rule — a TTL (time to live) or a policy like LRU (evict the least recently used entry) — otherwise the cache grows forever. And watch for a cache stampede: when a hot key expires, hundreds of requests miss at once and all hammer the database simultaneously.

When one database can't hold or serve all the data, split it into shards, each on its own machine. Range-based sharding (e.g. users A–M on shard 1, N–Z on shard 2) keeps ranges queryable together but can create uneven load. Hash-based sharding (hash the key, mod by shard count) spreads load evenly but makes range queries expensive. Directory-based sharding keeps an explicit lookup table of key → shard, which is flexible but adds a lookup hop.

A hot key is a single key (a viral post, a celebrity account) that gets so much traffic it overwhelms its one shard, no matter how many shards exist. Fixes include splitting the key's traffic further (e.g. appending a random suffix and merging results) or caching it aggressively.

Replication keeps multiple copies of the same data for durability and read scale. In leader–follower replication, writes go to the leader and are streamed to followers; reads can go to either, trading freshness for read capacity. Replication lag is the delay before a follower catches up — read a follower during lag and you can see stale data.

Quorum-based systems make this tunable: with N replicas, a write must be acknowledged by W of them, and a read must query R of them. If W + R > N, every read overlaps with the latest write on at least one replica, guaranteeing you see it — this is strong consistency via quorum overlap, not via a single leader.

Every network call needs a timeout — otherwise a slow dependency can hold a thread (or a connection) forever, and enough stuck threads can take down a healthy service. A retry (ideally with exponential backoff and jitter) handles transient failures, but blind retries can turn one slow dependency into a traffic avalanche.

A circuit breaker tracks failures to a dependency: in the Closed state, calls go through normally; once failures cross a threshold, it trips Open and fails fast without even calling the dependency, giving it time to recover; after a cooldown it goes Half-Open and lets one test call through — success closes the circuit, failure re-opens it. Back-pressure is the general idea behind all of this: a system tells its upstream "slow down" (via a bounded queue, a 429 response, or dropping the connection) instead of silently accepting more work than it can do and collapsing.

An operation is idempotent if doing it once has the same effect as doing it many times. By the HTTP spec, GET, PUT, and DELETE are idempotent — sending the same PUT twice leaves the resource in the same state. POST is not — sending "create an order" twice, naively, creates two orders. Since retries are unavoidable (timeouts don't tell you whether the server actually finished), clients attach a unique idempotency key to unsafe operations, and the server stores "I already handled key X, here's that same response" instead of repeating the side effect.

Statelessness means a server keeps no client-specific state between requests — session data lives in a token (e.g. a JWT) the client holds, or in a shared store (e.g. Redis) any server instance can read. That's what makes horizontal scaling and load balancing trivial in Chapter 1: any server can handle any request.

REST models the world as resources named by nouns (/orders, /orders/42), not verbs (never /getOrder). The HTTP method says what to do to that resource: GET read, POST create (or trigger a non-idempotent action), PUT replace fully, PATCH update partially, DELETE remove.

The status code is part of the contract, not decoration: 200 OK, 201 Created (with a Location header), 204 No Content, 400 Bad Request (malformed input), 401 Unauthorized (no/bad credentials), 403 Forbidden (authenticated but not allowed), 404 Not Found, 409 Conflict (e.g. duplicate), 422 Unprocessable Entity (well-formed but semantically invalid), 429 Too Many Requests, 500 Internal Server Error, 503 Service Unavailable. A good error contract returns a consistent JSON shape (e.g. {"error":{"code":"...","message":"..."}}) every time, not a different shape per endpoint.

// pick a scenario above

APIs need versioning because clients update on their own schedule: a URI version (/v2/orders) is explicit and cacheable; a header version (Accept: application/vnd.api+json;version=2) keeps URLs stable but is easy to miss. Either way, never break a version that clients already depend on — add a new one instead.

Offset pagination (?offset=20&limit=10) is simple but drifts under concurrent writes: if an item is inserted before page 2 is fetched, page 2 can repeat or skip a row. Cursor (keyset) pagination (?after=<last_id>&limit=10) anchors to a stable position and doesn't drift, and it's cheaper for the database at high offsets. Filtering belongs in query parameters (?status=shipped), never in the path. HATEOAS means responses include links to the actions available on that resource (e.g. a cancel link on an order that can still be cancelled) so clients discover capabilities instead of hardcoding URLs — the most-cited but least-implemented REST constraint in practice.

An API contract defines the shape of requests and responses (e.g. an OpenAPI spec); a data contract defines the shape of stored or transmitted data (e.g. a database schema or an event schema). Both need a compatibility rule so producers and consumers can change independently. A change is backward compatible if new code can still read data written by old code (a new consumer handles old messages). It's forward compatible if old code can still read data written by new code (an old consumer survives new messages, usually by ignoring fields it doesn't recognize).

Safe changes: adding an optional field, adding a new enum value if consumers already ignore unknown ones, loosening a validation rule. Breaking changes: removing a field still in use, renaming a field, changing a field's type, or adding a new required field with no default. Contract testing (e.g. a consumer-driven contract, or a schema registry that rejects incompatible changes) catches these breaks in CI, before a deploy — not after a downstream service starts throwing errors in production.

An event is a fact about something that already happened (OrderShipped) — the producer doesn't know or care who reacts to it; any number of consumers can subscribe. A command tells one specific receiver to do something (ShipOrder) — it implies an expected outcome and usually one intended handler. A notification is a lightweight event carrying little or no payload (just "something changed, go fetch it if you care") — cheap to send, but every consumer pays a round trip to find out what changed.

Whichever shape you pick, the message's schema evolves over time just like an API's, and the same compatibility rules from Chapter 12 apply — plus one more: put a version identifier in every event so consumers can tell old shapes from new ones, and never silently repurpose a field.

Most message brokers give you at-least-once delivery by default: a message might be delivered more than once (after a retry, a rebalance, or a crash before an acknowledgment), but never zero times. Exactly-once processing — as an observed effect — is achieved on top of at-least-once delivery by making consumers idempotent (the same idempotency-key idea from Chapter 9: track processed message IDs and skip duplicates), not by a broker magically deduplicating for you.

The classic bug is the dual-write problem: a service writes to its database, then separately publishes an event — but a crash between those two steps leaves the database changed with no event ever sent (or vice versa). The outbox pattern fixes this: write the business row and an outbox row in the same database transaction (so both commit or neither does), then a separate relay process reads the outbox and publishes to the broker, retrying until it succeeds. This guarantees the event is eventually published if and only if the transaction committed.

Worked example: design a URL shortener. Below is the order that turns a vague prompt into a defensible design — click each stage.

Cheat sheet