Load Balancers: How They Work, Algorithms, Health Checks and Stateless Services

In article 07 we worked out that Pixly needs about 18 application servers to carry its evening peak and survive the loss of a data centre. That raises an obvious question: users type one address, so how do their requests end up spread across 18 machines? And when one of those machines crashes at 9 p.m., who stops sending it traffic?

The answer is a load balancer: a component that sits in front of a group of servers, accepts every request, and forwards each one to a healthy server. It is usually the first piece of infrastructure a growing system adds, and it only works if the servers behind it are designed for it. This article covers both sides.

• What problems a load balancer solves.
• Stateless versus stateful services, and why statelessness makes scaling easy.
• How a load balancer forwards traffic, and the difference between Layer 4 and Layer 7.
• What a reverse proxy is, and how it relates to a load balancer.
• Routing algorithms: round robin, weighted, least connections, least outstanding requests, random choices and hashing.
• Health checks, connection draining and safe deployments.
• Sticky sessions: how they work and why they are usually a last resort.
• Making the load balancer itself highly available, and how autoscaling fits in.

A load balancer solves four problems at once:

1. Capacity. One server can only handle so many requests. Many servers behind one address can handle many more. This is horizontal scaling.
2. Availability. If one server fails, the load balancer stops sending it requests. Users barely notice.
3. Maintenance and deployment. You can take servers out of rotation, update them and put them back, one at a time, with no downtime.
4. A single, stable entry point. Clients know one address. Behind it, servers can come and go.

                 Clients                    |                    v          +--------------------+          |   Load balancer    |   one public address          +--------------------+            |       |       |            v       v       v         App 1   App 2   App 3     identical, stateless            \       |       /             v      v      v          Database, cache, object storage (shared state)

A load balancer can send any request to any server only if any server can handle any request. That property is called being stateless.

A stateless service keeps no information about a client between requests in its own memory or local disk. Everything it needs either arrives with the request (an authentication token, the request body) or is fetched from a shared store that every server can reach (a database, a cache, object storage). The Twelve-Factor App guidelines express the same idea: processes should be stateless and share nothing, with persistent data kept in a backing service (Twelve-Factor App: Processes).

A stateful service keeps client-specific data locally: a shopping cart in a server's memory, an uploaded file on its disk, a user's game state in a process. Requests from that client must then return to that exact server.

Local caches are an exception that is usually fine: a server may keep a copy of popular data in memory to go faster, as long as losing it only costs performance, not correctness. The test is: if this server vanished right now, would any user lose data or be logged out? If not, the service is stateless enough.

Statelessness is what makes horizontal scaling, autoscaling, rolling deployments and quick recovery cheap. That is why "keep the application tier stateless" is one of the most common sentences in good system-design answers.

At its simplest:

1. The client connects to the load balancer's address (found through DNS, article 03).
2. The load balancer picks a target server from a pool, using a routing algorithm.
3. It forwards the request (or the whole connection) to that target.
4. It relays the response back to the client.

Managed cloud load balancers spread themselves across several availability zones and route to targets in those zones. AWS describes Elastic Load Balancing as distributing incoming traffic across multiple targets, such as EC2 instances, containers and IP addresses, in one or more Availability Zones, and routing only to healthy targets (How Elastic Load Balancing works). Google Cloud and Azure offer equivalent services, and NGINX, HAProxy and Envoy are widely used software load balancers you can run yourself (NGINX HTTP load balancing).

"Layer" refers to the networking model. You need only two ideas.

AWS describes its Application Load Balancer as functioning at the application layer and routing based on the content of the request (What is an Application Load Balancer?). Google Cloud likewise offers both proxy-based application load balancers and network load balancers (Cloud Load Balancing overview).

When to use which: use Layer 7 for web and API traffic, where routing by path (/api/* to API servers, /admin/* to admin servers) and TLS termination are valuable. Use Layer 4 when you need raw speed, very high connection counts, non-HTTP protocols, or want to pass encrypted traffic straight through.

A reverse proxy is a server that sits in front of your servers and handles client requests on their behalf. Clients talk to the proxy; the proxy talks to the servers. (A forward proxy is the opposite: it sits in front of clients, for example a corporate proxy that employees' browsers go through.)

A Layer 7 load balancer is a reverse proxy that spreads load. Reverse proxies also commonly:

• terminate TLS and manage certificates;
• compress responses;
• cache responses (a CDN is a geographically distributed reverse proxy, article 10);
• buffer slow clients, so application servers are not tied up by a slow mobile connection uploading a body;
• enforce request size limits, timeouts and rate limits;
• hide the internal layout of your system.

In interviews, "load balancer" and "reverse proxy" are often used loosely for the same box at the edge. That is fine, as long as you can say what the box does.

Send request 1 to server A, request 2 to B, request 3 to C, request 4 to A again, and so on. Simple and fair when every request costs about the same and every server is the same size.

Like round robin, but a server with weight 2 gets twice as many requests as one with weight 1. Useful when servers differ in size, or when canary releasing a new version to a small share of traffic.

Send each new request to the server with the fewest active connections or in-flight requests. This adapts automatically when some requests are much slower than others, because a server stuck on slow requests accumulates connections and stops receiving new ones.

Pick a server at random. Surprisingly decent at scale, and it needs no shared state between load balancer instances. A well-known refinement is to pick two servers at random and send the request to the less loaded of the two, which avoids most of the imbalance of pure random choice while staying cheap.

Compute a hash of something in the request (client IP, user id, a URL) and use it to choose the server, so the same key always goes to the same server. This gives affinity, useful when servers hold something valuable per key, such as a cache shard. Plain hash mod N remaps almost every key when N changes; consistent hashing moves only a small fraction. Articles 16 and 32 explain consistent hashing in depth.

This small Java program models 12 requests arriving at the same moment for 3 servers. Every third request is expensive (9 units of work); the rest cost 1. Round robin happens to line up with that pattern and sends every expensive request to the same server. The second strategy sends each request to the server with the least work outstanding, a simplified model of least-outstanding-requests.

// File: BalancerDemo.javaimport java.util.Arrays;public class BalancerDemo {    public static void main(String[] args) {        int[] costs = {9, 1, 1, 9, 1, 1, 9, 1, 1, 9, 1, 1};        int servers = 3;        int[] roundRobin = new int[servers];        for (int i = 0; i < costs.length; i++) {            roundRobin[i % servers] += costs[i];        }        int[] leastWork = new int[servers];        for (int cost : costs) {            int best = 0;            for (int s = 1; s < servers; s++) {                if (leastWork[s] < leastWork[best]) {                    best = s;                }            }            leastWork[best] += cost;        }        System.out.println("Total work: " + Arrays.stream(costs).sum());        System.out.println("Round robin, work per server: " + Arrays.toString(roundRobin)                + "  busiest = " + Arrays.stream(roundRobin).max().getAsInt());        System.out.println("Least work,  work per server: " + Arrays.toString(leastWork)                + "  busiest = " + Arrays.stream(leastWork).max().getAsInt());    }}

Total work: 44Round robin, work per server: [36, 4, 4]  busiest = 36Least work,  work per server: [13, 19, 12]  busiest = 19

With round robin, one server does 36 units while the others idle at 4; the slowest request finishes at time 36. With the adaptive strategy the busiest server has 19, almost twice as fast overall. Real traffic is rarely this neatly aligned, but uneven request costs are normal (a search query versus a profile lookup), which is why adaptive algorithms are a common default for API traffic.

A load balancer must know which servers can take traffic. It finds out with health checks.

• Active checks: the load balancer calls an endpoint such as GET /health on each server every few seconds. After a number of consecutive failures (say 3), the server is marked unhealthy and removed; after a number of successes, it is added back.
• Passive checks: the load balancer watches real traffic. If a server returns errors or times out repeatedly, it is ejected for a while.

What should /health actually check?

• A shallow check confirms the process is up and can serve: it responds, it has finished starting, it is not out of memory.
• A deep check also verifies dependencies: can it reach the database, the cache, the payment provider?

Deep checks sound safer but carry a serious risk. If the shared database has a brief problem, every server's deep check fails at once, the load balancer removes every server, and a partial problem becomes a total outage. The Amazon Builders' Library article on health checks discusses exactly this tension, and recommends that automation remove a single bad server but keep serving if the whole fleet appears unhealthy, since then the health check itself is the more likely culprit (Implementing health checks). Some load balancers implement this "fail open" behaviour.

A practical rule: health checks should test what is local to the server. Dependency problems are better handled by timeouts, circuit breakers and degraded responses (article 23), and by alerts.

Container platforms split health into two questions. Kubernetes, for example, uses a liveness probe to decide when to restart a container, and a readiness probe to decide whether it should receive traffic, removing it from service endpoints without restarting it when the probe fails (Liveness, Readiness, and Startup Probes). The distinction is useful everywhere:

• Not ready: "don't send me traffic right now" (still starting, warming a cache, temporarily overloaded).
• Not live: "I am broken; restart me" (deadlocked, out of memory).

When you remove a server (to deploy, scale down or replace it), requests already in progress should finish. Load balancers support connection draining (also called deregistration delay): the server stops receiving new requests but keeps serving existing ones for a grace period, then is removed.

A rolling deployment uses this:

1. Mark one server (or a small batch) as draining.
2. Wait for in-flight requests to finish.
3. Deploy the new version.
4. Wait until its readiness check passes.
5. Return it to the pool, then repeat for the next batch.

Capacity drops slightly during each step, so deploy in small batches and not at peak if you are near your limit. For long-lived WebSocket connections (article 05), draining means asking clients to reconnect elsewhere, gradually.

Sticky sessions (session affinity) make the load balancer send all requests from one client to the same server, usually with a cookie the load balancer sets, or by hashing the client's IP address.

Why people use them: the application keeps session state in memory and was never made stateless. Stickiness makes it "work" with several servers.

Why they are usually a last resort:

• Uneven load. A few heavy users can pile onto one server.
• Fragility. When that server dies or is redeployed, its users lose their state.
• Scaling friction. New servers receive only new users, so adding capacity helps slowly.
• IP-based stickiness breaks when many users share one IP (offices, mobile carriers) or when a user's IP changes.

Stickiness is reasonable when the affinity is a performance optimisation, not a correctness requirement: for example, routing a user to the server that has their data warm in a local cache, where losing that server only costs a cache miss. And WebSocket connections are inherently sticky for their lifetime, because the connection itself lives on one server.

If every request goes through one load balancer, it becomes the single point of failure you were trying to remove. Ways to avoid that:

• Managed load balancers run as a redundant fleet across zones, behind one name; you never see the individual nodes.
• Self-managed pairs: two load balancer machines, one active and one standby, sharing a floating IP address that moves to the standby if the active one fails.
• Several load balancer addresses in DNS, so clients can use any of them.
• Anycast: the same IP address announced from many locations, with the network routing each client to the nearest healthy one (used by CDNs and global load balancers).

Across regions, load balancing is often done by DNS-based or anycast global load balancing, which sends users to the nearest healthy region (articles 03 and 24).

• At the edge, internet-facing, in front of web and API servers.
• Internally, between services: service A reaches service B through an internal load balancer.
• In the client: instead of a central load balancer, a client library looks up the list of healthy instances (from a service registry) and picks one itself. This "client-side load balancing" is common inside microservice systems and service meshes (article 26).
• In front of databases: read replicas behind a load balancer or a database proxy (article 15).

A load balancer pairs naturally with autoscaling: a controller that adds servers when load rises and removes them when it falls, registering and deregistering them with the load balancer. Typical signals are CPU utilisation, requests per target, or queue length.

Points to remember:

• New servers take time to start and warm up. Autoscaling reacts in minutes, not seconds, so it does not save you from a sudden spike (the flash sale in article 06). Use scheduled scaling before known events, and queues or load shedding for sudden ones.
• Keep a minimum capacity that survives the loss of a zone at peak (article 07).
• Autoscaling only works for stateless servers. That is another reason to keep them stateless.

                       Internet                          |                    DNS (one name)                          |              +-----------------------+              |  L7 load balancer     |  managed, spans 3 zones              |  TLS termination      |  health checks every 10 s              +-----------------------+               |                     |        /api/* |                     | /admin/*               v                     v   +-----------+-----------+     Admin servers (2)   |  API servers x 18     |   |  6 per zone, stateless|   +-----------------------+               |   Shared state: Redis (sessions, cache), PostgreSQL, object storage

Decisions and reasons:

• Layer 7, because we route by path and want TLS terminated in one place.
• Least outstanding requests, because feed requests and upload-initiation requests cost different amounts.
• Shallow health checks (/health returns OK if the process is serving and warmed up), so a database hiccup does not eject all 18 servers.
• No sticky sessions: sessions live in Redis, so any server can serve any user.
• Rolling deploys in batches of 3 (one per zone), with draining.
• Photos themselves do not pass through this load balancer; they are uploaded to and served from object storage behind a CDN (article 10). Keeping large transfers off the API path is a capacity decision in its own right.

• Adding servers without making them stateless, then discovering logouts and lost carts.
• Using sticky sessions to paper over state in memory.
• Health checks that call the database, payment provider and three other services.
• Forgetting that the load balancer itself needs redundancy.
• Sending large file uploads and downloads through the API servers.
• Expecting autoscaling to handle a spike that arrives in seconds.

• Put a load balancer in front of any tier with more than one server, and say "the servers are stateless; session state is in a shared store".
• Mention health checks and that the load balancer is redundant across zones.
• If request costs vary, prefer least-outstanding-requests over round robin, and say why.
• Only bring up L4 versus L7 or hashing if it matters for the design (non-HTTP protocols, cache affinity, WebSockets).

• Next: Caching in System Design: Patterns, Invalidation, Eviction and Redis, the next block, takes repeated reads off the database.
• Later: API gateways and service discovery build on load balancing (article 26), and multi-region traffic steering is in article 24.