Scalability Fundamentals

Vertical vs horizontal scaling

Vertical scaling (scale up): give the existing machine more resources — more CPU, RAM, faster disk. Horizontal scaling (scale out): add more machines and split the load across them.

	Vertical	Horizontal
How	Bigger single machine	More machines
Ceiling	Hard limit (biggest instance a cloud provider offers, physical hardware limits)	Effectively unbounded (add more nodes)
Downtime to scale	Usually requires a restart/resize	Add/remove nodes with zero downtime
Single point of failure	Yes — one machine, one failure domain	No, if properly load-balanced and stateless
Cost curve	Superlinear near the top end (the fastest CPUs/most RAM cost disproportionately more per unit)	Roughly linear (N commodity machines ≈ N × cost of one)
Complexity	Low — nothing about the application changes	Higher — requires statelessness, load balancing, data partitioning at some point

Vertical scaling is the right first move in almost every real system and every interview: it’s free complexity-wise, and a lot of workloads simply never outgrow a large single machine. The point where it stops working is where the interview actually gets interesting.

The evolution path

This is the walkthrough almost every system design interview expects, unprompted, as the opening few minutes: start with one server, and narrate what breaks next as load grows.

flowchart TD
    A["1. Single server\n(app + DB on one box)"] --> B["2. Separate app and DB servers\n(scale each independently, DB gets dedicated resources)"]
    B --> C["3. Load balancer + multiple app servers\n(horizontal scaling of the stateless tier)"]
    C --> D["4. Add a cache\n(absorb read load, cut DB round trips)"]
    D --> E["5. Add read replicas\n(scale DB reads horizontally)"]
    E --> F["6. Shard the database\n(scale DB writes by splitting data across nodes)"]
    F --> G["7. CDN + async processing + microservices\n(offload static content, decouple slow work via queues)"]

Walking through each step and why it’s the next bottleneck:

Single server: app and DB on one box. Fine until either CPU/RAM saturates or the DB and app start competing for the same resources.
Split app and DB onto separate machines: immediately doubles effective capacity for both, and lets each be scaled/tuned independently (DB usually needs more RAM/IO, app tier needs more CPU).
Load balancer + multiple app servers: the app tier is (once made stateless — see below) the easy part to scale horizontally; a load balancer distributes requests across N identical app instances.
Cache: the database becomes the next bottleneck — most workloads are read-heavy, so a cache (see Caching Strategies) absorbs a large fraction of reads before they ever reach the DB.
Read replicas: for read traffic the cache can’t absorb (or as a second line of defense), replicate the DB and route reads across replicas — scales read capacity, not write capacity (all writes still go to the primary).
Sharding: once write volume or total data size outgrows a single primary, split the data itself across multiple DB nodes by some key (see Database Sharding & Replication) — this is the point where the database, not the app tier, becomes the thing being horizontally scaled, and it’s meaningfully harder (cross-shard queries, rebalancing, choice of shard key all become real design problems).
CDN, async processing, service decomposition: push static/cacheable content to the edge (see CDN & Content Delivery), move slow non-critical work off the request path via queues (see Message Queues & Event-Driven Architecture), and eventually split the app itself into independently-scalable services (see Microservices vs Monolith) once different parts of the system have genuinely different scaling profiles.

The database is almost always the real bottleneck, not the app tier — app servers are cheap to clone because (once stateless) they hold no unique data; a database holds the one copy of the truth, so scaling it is a fundamentally harder problem, which is why steps 4-6 exist specifically to delay or distribute that difficulty.

Statelessness: the precondition for horizontal scaling

Horizontal scaling only works cleanly if any instance can handle any request. That requires the app tier to hold no unique, non-reproducible state in local memory or on local disk — session data, uploaded files, in-flight computation state all need to live somewhere shared (a DB, a distributed cache, object storage) instead of on the specific server that happened to handle the first request.

What breaks with server-local state

sequenceDiagram
    participant C as Client
    participant LB as Load Balancer
    participant S1 as Server 1 (has session in local memory)
    participant S2 as Server 2 (no knowledge of the session)

    C->>LB: POST /login
    LB->>S1: route (round robin)
    S1->>S1: store session in local memory: {user: 42, cart: [...]}
    S1-->>C: 200 OK, Set-Cookie: session_id=abc

    C->>LB: GET /cart (session_id=abc)
    LB->>S2: route (round robin picks a different server)
    S2->>S2: look up session_id=abc in local memory
    Note over S2: not found - S2 never saw this session
    S2-->>C: 401 Unauthorized (looks logged out, cart is "empty")

The user just got randomly logged out because the load balancer, correctly doing its job of spreading load, happened to route their second request to a different server than their first. The naive “fix” is sticky sessions (pin the client to server 1) — but that just relocates the problem: server 1 becomes a soft single point of failure for that user, and the LB can no longer freely rebalance load (see Load Balancing for why this actively fights horizontal scaling).

The real fix: externalize the state.

flowchart LR
    S1[App Server 1] --> Shared[("Shared session store\n(Redis / DB)")]
    S2[App Server 2] --> Shared
    S3[App Server 3] --> Shared

With session state in a shared store, every app server is interchangeable — any instance can serve any request, the load balancer can route by pure load with no affinity requirement, and servers can be added, removed, or crashed and replaced without losing anyone’s session. This is precisely what makes the app tier the “easy” part of the evolution path above: once stateless, it just needs more identical copies.

The CAP theorem

For a distributed data store, you can only fully guarantee two of three properties simultaneously:

Consistency (C): every read sees the most recent write (or an error) — all nodes return the same, latest value at the same time.
Availability (A): every request to a non-failed node receives a (non-error) response — no request just hangs or gets refused.
Partition tolerance (P): the system keeps operating even when network messages between nodes are dropped or delayed (a network partition).

The theorem’s actual claim is narrower than it’s often stated: partitions will happen in any real distributed system (networks are unreliable — cables get cut, switches fail, packets get dropped) — partition tolerance isn’t really an optional design choice, it’s a fact about physical networks you must accept. So the real, live tradeoff is only exposed during an actual partition: when nodes can’t talk to each other, do you sacrifice consistency (keep answering, possibly with stale/conflicting data) or availability (stop answering on the side that can’t confirm it has the latest data)?

Concrete partition scenario

flowchart TD
    subgraph "Before partition"
    C1["Node A (leader)"] <--> C2["Node B (replica)"]
    end

sequenceDiagram
    participant Client1 as Client (partition A side)
    participant A as Node A (was leader)
    participant B as Node B (replica, cut off)
    participant Client2 as Client (partition B side)

    Note over A,B: network partition - A and B can no longer reach each other

    Client1->>A: write x = 5
    A-->>Client1: ack (write succeeded on A)

    Client2->>B: read x
    alt CP choice (favor Consistency)
        B-->>Client2: error / unavailable (can't confirm this is the latest value)
    else AP choice (favor Availability)
        B-->>Client2: x = 3 (last known value before partition - stale, but a response)
    end

CP system (e.g. HBase, MongoDB in its default majority-write config, ZooKeeper, most consensus-based systems like etcd): during the partition, node B refuses to answer (or a quorum-based system refuses writes on the minority side) rather than risk returning stale or conflicting data. Correctness is preserved; availability on the cut-off side is not.
AP system (e.g. Cassandra, DynamoDB in its default config, CouchDB): node B keeps answering with whatever it has, even though it may be stale relative to A. Availability is preserved everywhere; the client on B’s side might see an old value, and the two sides may have accepted conflicting writes that need reconciling once the partition heals (see eventual consistency / conflict resolution, e.g. last-write-wins or vector clocks).

It’s a spectrum, not a binary

In practice, systems don’t sit at a pure CP or pure AP extreme, and the choice is often tunable per operation, not fixed for the whole system:

Many systems let you choose the consistency level per request — DynamoDB offers both eventually-consistent reads (cheaper, possibly stale) and strongly-consistent reads (more expensive, always latest) on the same table.
Quorum-based systems (Cassandra, Dynamo-style databases) let you tune N (replicas), W (writes required to ack), and R (reads required to ack) per operation — W + R > N gives strong consistency, W + R <= N trades it for lower latency/higher availability, all on the same cluster.
When there’s no partition (the common case — partitions are rare, not constant), a well-designed system can actually deliver both C and A simultaneously; CAP only forces a real choice during the partition window itself.
A more complete framing (PACELC) extends this: if Partitioned, choose A or C; Else (normal operation, no partition), choose Latency or Consistency — because even without a partition, strong consistency (waiting for all/quorum replicas to confirm) costs latency versus returning a possibly-stale local value faster.

Interview framing: name-dropping “CAP theorem” isn’t the signal — explaining what happens during a specific partition scenario for the system being designed is. For a payments ledger, argue for consistency (a stale balance is a correctness bug). For a social media like-count or a presence indicator (“user is online”), argue for availability (a slightly stale count is fine, refusing to load the page is not).

Amdahl’s law and diminishing returns

Amdahl’s law quantifies why throwing more parallel workers at a problem has diminishing returns: if a fraction P of a task can be parallelized and the rest, (1-P), is inherently serial, the maximum possible speedup with N parallel workers is:

speedup(N) = 1 / ((1 - P) + P/N)

As N → ∞, speedup converges to 1 / (1 - P) — bounded purely by the serial fraction, no matter how many workers you add.

Serial fraction (1-P)	Max speedup as N → ∞
50%	2x
10%	10x
5%	20x
1%	100x

The practical takeaway for system design: adding more app servers, more shards, or more worker threads only helps up to the point where some serial bottleneck (a single-writer database, a global lock, a single message queue partition, a coordination step every request must pass through) dominates. Identifying that bottleneck — the part of the pipeline that can’t be parallelized away — is usually more valuable in an interview than proposing “add more machines” as a generic answer. This is the same underlying reason sharding a database (removing the single-writer bottleneck) has a bigger structural impact than just adding more read replicas (which only ever parallelizes the already-parallel read path).

Capacity estimation (back-of-envelope math)

A standard interview skill: given a rough scale (daily active users, DAU), estimate the load the system needs to handle — QPS and storage — to sanity-check the design and justify choices like “we need sharding” or “a single DB is fine.” Precision doesn’t matter; being within the right order of magnitude and showing the reasoning does.

Worked example: design a service where each of 10 million DAU posts on average 2 times per day, and each post is read (viewed in a feed) 50 times on average.

Write QPS:

10,000,000 users × 2 posts/day = 20,000,000 writes/day
20,000,000 / 86,400 seconds/day ≈ 231 writes/sec (average)

Peak traffic is never uniform across the day — a common rule of thumb is peak ≈ 2-3x average:

231 × 3 ≈ 700 writes/sec (peak)

Read QPS:

20,000,000 posts/day × 50 reads/post = 1,000,000,000 reads/day
1,000,000,000 / 86,400 ≈ 11,600 reads/sec (average)
peak ≈ 11,600 × 3 ≈ 35,000 reads/sec

Read:write ratio here is roughly 50:1 — this alone justifies aggressive read-side caching (see Caching Strategies) and read replicas over trying to scale writes.

Storage:

Each post ≈ 280 bytes text + 100 bytes metadata (id, timestamp, user_id, etc.) ≈ 400 bytes
20,000,000 posts/day × 400 bytes ≈ 8 GB/day of new post data
Over 5 years: 8 GB × 365 × 5 ≈ 14.6 TB

If posts commonly include media (say 20% include a 2 MB average image/video reference stored separately in object storage): 20,000,000 × 0.20 × 2 MB = 8 TB/day in blob storage alone — several orders of magnitude bigger than the text/metadata, which is exactly the kind of finding that should steer the design (blob storage + CDN for media, separate from the metadata database).

What this estimation exercise is actually for in an interview: it justifies concrete decisions downstream — “35K reads/sec means we need a cache in front of the DB and probably read replicas,” “8 TB/day of media means object storage + CDN, not storing blobs in the primary DB,” “700 writes/sec on a single well-indexed primary is comfortably within a single modern DB’s capability, so we don’t need to shard for writes yet.” Skipping the math and jumping straight to “we’ll need sharding” is a common tell that the estimate wasn’t actually grounding the design.

Vertical scaling limits

Hardware ceiling: even the largest cloud instances top out (e.g. hundreds of vCPUs, a few TB of RAM) — there’s a real, discoverable maximum, unlike horizontal scaling’s “just add another node.”
Cost curve: price does not scale linearly with capacity near the top end — doubling from a mid-tier instance to the next tier up often costs more than double, because larger machines are lower-volume, more specialized hardware. At some point, N smaller machines are cheaper than 1 bigger one for the same aggregate capacity.
Single point of failure, structurally: no matter how large the one machine is, it’s still one failure domain — one hardware fault, one bad deploy, one OS panic takes down 100% of capacity. Horizontal scaling turns that into a partial-degradation problem (lose 1 of N servers, lose roughly 1/N of capacity) instead of a total outage.
Downtime to resize: vertically resizing typically means a restart (or at least a brief interruption) of that instance — horizontal scaling can add/remove capacity with zero downtime to existing instances.
In practice, the right answer is “both, in order”: right-size instances vertically first (cheap, no architectural change), and reach for horizontal scaling once vertical headroom or its cost curve stops making sense — not either/or.

Interview angles

The opening move of almost every system design interview: “let’s start with one server — what breaks first as traffic grows?” Be ready to narrate the full evolution path unprompted, and to justify why each step is the next bottleneck, not just list the steps.
“Why can’t we just add more app servers?” → if the app is stateless, you can, and should; if it isn’t, statelessness has to come first — be ready to give the concrete session-on-one-server failure mode.
“Talk to me about CAP theorem” → don’t just define the letters; walk through what happens to a specific read/write during a specific partition in the system being designed, and state which side you’d choose and why for that specific data (ledger balance vs. like count).
“How many servers/how much storage do we need?” → do the back-of-envelope math out loud: DAU → QPS (average, then ×2-3 for peak) → storage/day → storage over N years, and use the result to justify a design decision (cache, replicas, sharding, object storage) rather than leaving the numbers disconnected from the design.
“Why not just get a bigger machine?” → know the real ceiling (hardware limits, superlinear cost at the high end, still a single failure domain) — vertical scaling isn’t wrong, it’s just not infinite, and a senior answer explains where it stops paying off rather than dismissing it outright.

Notes

Explorer