Load Balancing

Why load balancers exist

A single server can only handle so much traffic and is a single point of failure. A load balancer sits in front of a pool of servers and distributes incoming traffic across them, giving two things at once: horizontal scalability (add servers, get more capacity) and fault tolerance (a dead server stops receiving traffic without the client noticing). Almost every non-trivial system design answer places one within the first few minutes — the interesting part is which kind, which algorithm, and where else in the architecture it belongs.

L4 vs L7

The layer refers to the OSI model layer the load balancer makes decisions at.

L4 (transport layer)

Operates on IP address and TCP/UDP port only — it never looks inside the packet payload. Typically implemented as fast packet forwarding (sometimes even in kernel space, e.g. Linux IPVS, or hardware) — it just routes connections, it doesn’t understand HTTP, headers, or cookies.

flowchart LR
    Client -->|"TCP SYN to VIP:443"| L4["L4 Load Balancer<br/>(routes by IP:port, no payload inspection)"]
    L4 -->|"forwards raw TCP stream"| S1[Server 1]
    L4 -.->|"forwards raw TCP stream"| S2[Server 2]
    L4 -.->|"forwards raw TCP stream"| S3[Server 3]

Extremely fast and low-overhead (no parsing above the transport header).
Protocol-agnostic — works for HTTP, gRPC, raw TCP, WebSocket, any TCP/UDP traffic identically.
Can’t make routing decisions based on content — no “route /api/* here, /static/* there,” no cookie-based stickiness, no header inspection.
Once a connection is routed to a backend, it typically stays with that backend for the connection’s lifetime (connection-level, not request-level, balancing).

L7 (application layer)

Terminates the connection, reads the actual HTTP request (method, path, headers, cookies, even body), and routes based on that content.

flowchart LR
    Client -->|"HTTP GET /api/orders/42"| L7["L7 Load Balancer<br/>(terminates TCP, parses HTTP)"]
    L7 -->|"path starts with /api/orders"| OrdersSvc[Orders Service pool]
    L7 -->|"path starts with /api/users"| UsersSvc[Users Service pool]
    L7 -->|"path starts with /static"| CDN[Static asset servers]

Content-aware routing: path-based (/api vs /static), header-based, cookie-based (sticky sessions), even A/B testing by routing a percentage of traffic to a canary version.
Can do things L4 can’t: SSL/TLS termination, request/response rewriting, compression, WAF (web application firewall) rules, retries on a failed backend for idempotent requests.
More overhead per request (parses the full HTTP request), and because it terminates the connection, it needs to open a new connection to the backend — two TCP connections per request instead of one.
Almost every public-facing HTTP load balancer in practice is L7 (NGINX, Envoy, AWS ALB, GCP HTTP(S) LB); L4 shows up more for raw TCP services, or as a cheap first tier in front of an L7 tier, or for non-HTTP protocols.

	L4	L7
Sees	IP + port	Full HTTP request (headers, path, cookies, body)
Speed	Faster, less overhead	Slower, more overhead
Routing granularity	Per-connection	Per-request
Protocol awareness	None (any TCP/UDP)	HTTP-specific (or needs a protocol-specific variant)
Examples	Linux IPVS, AWS NLB, hardware LBs (F5)	NGINX, HAProxy, Envoy, AWS ALB
Typical use	Raw throughput, non-HTTP protocols, first tier in front of L7	Content-based routing, most public web/API traffic

Load balancing algorithms

Algorithm	How it works	Pick it when
Round robin	Requests distributed in fixed rotation (1, 2, 3, 1, 2, 3…)	Backends are roughly identical in capacity and request cost is roughly uniform — the simplest safe default
Weighted round robin	Same rotation, but servers with higher weight get proportionally more requests	Backends have different capacities (e.g. mixed instance sizes during a rolling upgrade)
Least connections	Route to whichever backend currently has the fewest active connections	Request durations vary a lot (some requests are slow, some fast) — round robin would overload a server stuck with several slow requests
Weighted least connections	Least connections, adjusted by server capacity weight	Same as above, plus heterogeneous server capacity
IP hash	Hash the client IP to deterministically pick a backend	Need the same client to consistently land on the same backend without a session store (crude sticky sessions)
Consistent hashing	Hash the request key onto a ring shared with backend positions; only a small fraction remaps when a backend joins/leaves	Backend set changes frequently (autoscaling) and you need key-to-backend stickiness without a full remap each time — see Consistent Hashing
Random	Pick a backend uniformly at random	Surprisingly close to round robin in practice; trivial to implement, no state needed
Least response time	Route to the backend with the best combination of active connections and recent response latency	Backend performance varies and you want to actively steer away from a slow (not just busy) node

Round robin and least connections cover the vast majority of real deployments. Consistent hashing matters specifically when the backend pool churns (autoscaling groups, container rescheduling) and you still want a given key routed to the same backend most of the time — e.g. load balancing to a fleet of cache nodes, where you want cache hits to keep landing on the node that has the data.

Health checks

A load balancer is only as good as its view of which backends are actually healthy — routing to a dead server is worse than routing nowhere (client waits for a timeout instead of getting a fast failure or a retry elsewhere).

flowchart TD
    LB[Load Balancer] -->|"active: periodic GET /health every 5s"| S1[Server 1: healthy]
    LB -->|"active: periodic GET /health every 5s"| S2[Server 2: no response x3 -> marked unhealthy]
    LB -.->|"traffic withheld"| S2
    LB -->|"passive: observes real traffic"| S3["Server 3: 90% of last 20 requests\n returned 500 -> marked unhealthy"]
    LB -.->|"traffic withheld"| S3

Active health checks: the load balancer proactively pings each backend on an interval (e.g. GET /health every few seconds), independent of real traffic. A backend that misses N consecutive checks is pulled from rotation; once it passes M consecutive checks again it’s added back (the asymmetric N/M thresholds avoid flapping a backend in and out over a single blip).
Passive health checks: inferred from real traffic — if a backend’s error rate or timeout rate crosses a threshold over a rolling window, it’s marked unhealthy, no separate probe needed. Catches problems that only manifest under real request patterns (an endpoint failing only for a certain payload shape a synthetic health check wouldn’t reproduce).
Production systems typically run both: active checks catch a backend that’s fully down even with zero live traffic hitting it; passive checks catch degraded-but-technically-up backends faster than an active probe interval might.
The health check endpoint itself matters: a “liveness” check (process is running) is weaker than a “readiness” check (process can actually serve — e.g. it has a working DB connection); a common failure mode is a backend that’s alive but can’t reach its database, still passing a naive liveness probe while failing every real request.

Sticky sessions

Some applications keep per-user state in the memory of whichever server first handled that user (a shopping cart, a login session) — sticky sessions (a.k.a. session affinity) route that user’s subsequent requests back to the same backend, usually via a cookie the LB sets or via IP hash.

sequenceDiagram
    participant C as Client
    participant LB as Load Balancer
    participant S1 as Server 1
    participant S2 as Server 2

    C->>LB: request 1 (no session cookie)
    LB->>S1: route (round robin picks S1)
    S1-->>C: response + Set-Cookie: server=S1
    C->>LB: request 2 (cookie: server=S1)
    LB->>S1: route (sticky - always S1 now)
    Note over S1,S2: S1 goes down
    C->>LB: request 3 (cookie: server=S1)
    LB--xS1: S1 unreachable
    Note over C: session state on S1 is lost - user logged out / cart emptied

Why it fights horizontal scaling: sticky sessions couple a client to one specific server, which undermines the entire point of a server pool.

Losing that one server loses that user’s state — no other server has a copy.
Load balancing degrades to “roughly balanced” instead of truly balanced — a burst of users who all got assigned to the same backend early on will overload it while others sit idle, and the LB can’t rebalance without breaking those users’ sessions.
Adding or removing servers (autoscaling) means reshuffling which users map where, which is exactly the disruption stickiness was meant to avoid.

The fix: externalize the state — put session data in a shared store (Redis, a DB) that every backend can read, so any server can serve any request statelessly. This is the core idea behind statelessness as a scaling precondition, covered in depth in Scalability Fundamentals. Sticky sessions are a valid short-term patch (or genuinely necessary for things like WebSocket connections which are inherently long-lived and stateful at the transport level) but should be treated as a smell to design away from, not a default.

Global vs local load balancing

Local (a load balancer tier): distributes traffic across servers within one data center / region — everything covered above.
Global (DNS-based / GeoDNS / Anycast): distributes traffic across data centers or regions, before it ever reaches a local load balancer. A GeoDNS resolver answers a DNS query for api.example.com with the IP of the nearest (or healthiest) regional entry point based on the requester’s location; Anycast takes it further by advertising the same IP from multiple locations and letting network routing (BGP) deliver the packet to the nearest one. Concretely: a client in the EU resolving api.example.com gets back the eu-west region’s IP, a client in the US gets us-east’s — each then hits that region’s own local LB tier, which does the fine-grained per-request distribution described above.

Global LB reduces latency (route to the nearest region) and gives regional failover (a region-wide outage is routed around at the DNS/Anycast level, entirely above any single local LB tier, by simply no longer resolving to that region’s IP). This is the same principle a CDN operates on, one layer up the stack — see CDN & Content Delivery.

Load balancer high availability

The load balancer itself must not become the single point of failure it exists to eliminate. The standard pattern is an active-passive pair (or active-active, for L4 setups that support it) with a floating virtual IP (VIP):

flowchart TD
    Client -->|"traffic to VIP"| VIP{{"Virtual IP\n(owned by whichever LB is active)"}}
    VIP --> Active["LB 1 - ACTIVE"]
    Passive["LB 2 - PASSIVE\n(heartbeat monitoring LB 1)"] -.->|"health check / heartbeat"| Active
    Active --> Pool[Backend server pool]
    Passive -.->|"on missed heartbeats: takes over VIP, becomes ACTIVE"| VIP

Two (or more) LB instances run behind a shared virtual IP; only one holds the VIP and actively handles traffic at a time.
The standby continuously heartbeats the active instance; if it stops responding, the standby claims the VIP (via a protocol like VRRP/keepalived, or a cloud provider’s managed failover) and starts serving traffic — typically within seconds.
In cloud environments this is usually handled by a managed load balancer service (AWS ALB/NLB, GCP Load Balancing) which is itself already distributed and highly available under the hood — the active-passive pattern is what you’d reach for if building/self-hosting the LB tier (e.g. HAProxy/NGINX on VMs), and it’s worth knowing the mechanism even when the interview answer is “use the cloud provider’s managed LB and let them handle this layer.”
DNS-level redundancy (multiple A records, health-checked DNS failover) provides another layer above this for surviving a whole LB tier or region going down.

Where load balancers actually sit

Real architectures use load balancers at more than one tier — not just the obvious client-facing one.

flowchart TD
    Client --> GeoLB["Global LB (GeoDNS)"]
    GeoLB --> EdgeLB["L7 Load Balancer\n(client-facing, TLS termination, routing)"]
    EdgeLB --> GW["API Gateway"]
    GW --> LB1["Internal LB"]
    LB1 --> Orders1[Orders Service instance 1]
    LB1 --> Orders2[Orders Service instance 2]
    GW --> LB2["Internal LB"]
    LB2 --> Users1[Users Service instance 1]
    LB2 --> Users2[Users Service instance 2]
    Orders1 --> LB3["Internal LB"]
    LB3 --> Inv1[Inventory Service instance 1]
    LB3 --> Inv2[Inventory Service instance 2]

Client-facing: the LB everyone thinks of first — public internet traffic into the edge of the system.
Service-to-service (east-west traffic): in a microservices architecture (see Microservices vs Monolith), every service that calls another service through multiple instances needs the same load-balancing problem solved internally — which instance of the Orders service handles this call? This is often done via a lightweight LB colocated with each caller (a sidecar proxy, e.g. Envoy in a service mesh) rather than routing all internal traffic through one central internal LB tier, so that internal calls don’t pay an extra network hop through a shared chokepoint.
In front of stateful tiers: databases with read replicas, or a distributed cache cluster, often sit behind their own routing layer too (even if it’s client-library-based routing rather than a dedicated LB box) — same underlying problem (many backends, pick one) recurring at every tier.

Interview angles

Nearly every system design answer needs a load balancer within the first few minutes — place it without hesitation, then be ready to go deeper when asked.
“L4 or L7 here?” → default to L7 for HTTP/web traffic (content-based routing, TLS termination) and justify L4 only for raw throughput or non-HTTP protocols.
“How do you handle a server going down?” → active + passive health checks pulling it from rotation, plus the LB’s own HA (active-passive pair or managed service) so the LB itself isn’t the new single point of failure.
“Requests are slow only for some users” → could be sticky sessions pinning them to one overloaded backend; the fix is externalizing state so any backend can serve any request.
“How would this scale to multiple regions?” → global/GeoDNS load balancing on top of the regional LB tier, and note that this doubles as your regional failover story.
A strong answer distinguishes “where does the LB sit” (client-facing vs internal service-to-service vs in front of a data tier) rather than assuming one LB in the diagram covers the whole system.

Notes

Explorer