There comes a point in almost every successful project where one server simply isn’t enough. Traffic grows, requests pile up, and the machine that used to answer instantly starts to crawl. The obvious fix is to add more servers — but that immediately raises a question nobody asks until they hit it: when ten thousand people type your address, which server actually answers each of them?
That’s the job of a load balancer. It sits in front of your servers, takes every incoming request, and decides where to send it. Done well, your visitors never know there’s more than one machine back there — they just see a fast, reliable site. Let’s unpack how that works, why it matters, and the trade-offs you’ll run into along the way.
What load balancing actually is
Load balancing is the practice of spreading incoming work across multiple servers so that no single one gets overwhelmed. The piece of software (or hardware) that does the spreading is the load balancer.
Picture a busy bank with one teller and a line out the door. Painful. Now picture the same bank with five tellers and a single person at the entrance directing each customer to whichever teller is free. The line moves five times faster, and if one teller goes on break, the other four keep serving. The person at the door is your load balancer. The tellers are your servers.
In web terms, the load balancer is the public face of your site — the single address the outside world connects to. Behind it sit several identical backend servers (often called a “pool” or “farm”), each running a copy of your application. The load balancer’s whole job is to answer the question “who handles this one?” thousands of times per second.
┌────────────────┐
request ───────► │ LOAD BALANCER │
└───────┬────────┘
┌──────────────────┼──────────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Server A │ │ Server B │ │ Server C │
│ (copy of │ │ (copy of │ │ (copy of │
│ the app) │ │ the app) │ │ the app) │
└───────────┘ └───────────┘ └───────────┘The visitor connects to one address. The load balancer quietly fans that request out to whichever backend is the best choice right now. The reply travels back the same way. As far as the browser is concerned, it talked to a single, very fast server.
Load balancer vs reverse proxy
You’ll often hear these two terms in the same breath, and they overlap. A reverse proxy is any server that receives requests on behalf of one or more backends and forwards them along. A load balancer is a reverse proxy whose specific job is spreading traffic across several backends using some strategy. Put simply: every load balancer is a reverse proxy, but not every reverse proxy is balancing load. Many real tools do both roles at once.
Why one server isn’t enough
If a single machine could handle everything forever, none of this would exist. But servers have hard limits, and you bump into them in two ways.
The first is capacity. Every server has a ceiling — how much memory it has, how many requests its CPU can churn through per second, how many simultaneous connections it can hold open. Push past that ceiling and requests start queueing, response times balloon, and eventually the machine falls over. You can buy a bigger machine (that’s called scaling up), but there’s always a limit to how big one box can get, and big boxes get expensive fast.
The second is reliability. A single server is a single point of failure. If it crashes, reboots for an update, or its data center loses power, your entire site goes dark. There’s no plan B. One machine means one thing standing between you and an outage.
Load balancing solves both at once by letting you run many smaller, cheaper servers side by side — an approach called scaling out. Need more capacity? Add another backend to the pool. One server died? The load balancer notices and stops sending traffic to it while the others carry on. Your site stays up.
SCALE UP SCALE OUT
(one bigger machine) (more machines, balanced)
┌─────────────┐ ┌──┐ ┌──┐ ┌──┐ ┌──┐
│ │ │ │ │ │ │ │ │ │
│ BIG │ └──┘ └──┘ └──┘ └──┘
│ SERVER │ ▲ ▲ ▲ ▲
│ │ └────┴─┬──┴────┘
└─────────────┘ load balancer
limit: how big can limit: almost none — just
one box get? keep adding boxesHow a load balancer picks a server
So the load balancer has, say, three healthy backends and a request just landed. How does it choose? This is where algorithms come in. Each one is a different rule for deciding who’s next, and the right choice depends on what your servers and traffic look like.
Round robin
The simplest rule of all: just go in order. First request to Server A, next to B, next to C, then back to A, and around again. It’s predictable, it’s fair, and it needs no information about the servers at all.
request 1 → A
request 2 → B
request 3 → C
request 4 → A (back to the start)
request 5 → BRound robin works beautifully when all your servers are roughly equal in power and every request costs about the same amount of work. Its weakness shows when those assumptions break — if one request takes ten seconds and the next takes ten milliseconds, blindly rotating can leave one server buried while another sits idle.
Least connections
Instead of taking turns, send each new request to whichever server is currently handling the fewest open connections. The idea is that a server with fewer active connections is probably less busy, so it can take more work.
This handles uneven request times far better than round robin. If Server A is stuck on a few slow requests, its connection count stays high, so the balancer naturally routes new traffic to the less-loaded B and C. It’s a smarter default for most real-world apps where request cost varies.
Weighted variants
Both of the above have weighted versions for when your servers aren’t equal. Maybe Server A is a beefy machine and B is a smaller one. You give A a weight of 3 and B a weight of 1, and the balancer sends roughly three times as much traffic to A. This lets you mix hardware of different sizes in the same pool without overloading the weaker boxes.
IP hash
Here the balancer takes the visitor’s IP address, runs it through a hashing function, and uses the result to pick a server. The clever part: the same visitor always lands on the same server, because their IP always hashes to the same number. We’ll see in a moment why that’s sometimes exactly what you want.
There's no single 'best' algorithm
Round robin and least connections cover the vast majority of cases, and most setups start with one of them. Don’t agonize over the choice early on — pick a sensible default (least connections is a safe bet when request times vary), get your site running, and only reach for the fancier options once you have a real reason. Premature tuning here is a classic way to waste an afternoon.
The sticky session problem
Now for the gotcha that trips up almost everyone the first time they put a site behind a load balancer.
Imagine a user logs in. The server that handled the login stores their session — “this person is Jane Doe, she’s authenticated” — in its own memory. On their next click, the load balancer (using round robin, say) sends them to a different server, which has never heard of Jane. Suddenly she’s logged out for no reason. Refresh, and she might be logged in again, then out again — a maddening, flickering mess.
The problem is that the user’s data lived on one specific server’s memory, but requests are being spread across all of them. There are two clean ways out.
Sticky sessions (also called session affinity) tell the load balancer to keep sending the same user back to the same server for the duration of their visit — often using that IP-hash trick, or a small cookie. Jane always returns to the server that knows her. It works, but it’s a bit fragile: if that one server goes down, everyone pinned to it loses their session anyway, and your traffic can become unevenly distributed.
The better long-term fix is to make your servers stateless — store session data somewhere shared (a database or a fast in-memory store) that every backend can reach, instead of in any one server’s local memory. Now it doesn’t matter which server answers Jane’s next click; they all look up her session from the same shared place. This is the approach that scales cleanly, and it’s worth designing for from the start.
STICKY SESSIONS STATELESS + SHARED STORE
Jane ─┐ Jane ─┐
└─ always → Server B ├─ any server
(her session ▼ ▼ ▼
lives here) ┌───┐ ┌───┐ ┌───┐
│ A │ │ B │ │ C │
if B dies, Jane's └─┬─┘ └─┬─┘ └─┬─┘
session is gone └──────┼──────┘
▼
┌──────────────┐
│ shared store │ ← session lives here
└──────────────┘
any server can serve JaneHealth checks: knowing who’s alive
A load balancer is only useful if it doesn’t send traffic to dead servers. So every decent one runs health checks — it periodically pokes each backend to ask “are you okay?” and only routes real traffic to the ones that answer correctly.
A health check can be as simple as opening a connection to the server’s port, or as thorough as requesting a special URL (often something like /health) and checking that the response comes back with the right status. If a server fails a few checks in a row, the balancer marks it unhealthy and quietly stops sending it requests. When it recovers and starts passing checks again, it’s added back into rotation automatically.
┌──────────────┐
│ LOAD BALANCER│ every few seconds: "you alive?"
└──┬────┬────┬─┘
│ │ │
OK OK ✗ no response
▼ ▼ ▼
┌───┐┌───┐┌───┐
│ A ││ B ││ C │ ← C marked unhealthy,
└───┘└───┘└───┘ gets no traffic until it recoversThis is the mechanism that turns “we have several servers” into “we have a site that survives a server dying.” Without health checks, a crashed backend would keep receiving its share of requests, and every user unlucky enough to be routed there would hit an error. With them, failures become almost invisible to your visitors.
Where the load balancer lives
Load balancing isn’t one fixed thing — it can happen at different layers, and you’ll meet a few flavors.
A layer 4 balancer works at the transport level (the world of TCP/IP and port numbers). It makes its decision based purely on connection-level info, without looking inside the actual request. It’s fast and simple because it doesn’t read or understand the traffic — it just shovels connections to backends. If port numbers and the basic shape of network traffic feel hazy, our piece on IP addresses and ports lays that groundwork.
A layer 7 balancer works at the application level — it actually understands HTTP. Because it can read the request, it can make smarter decisions: send all requests for /images/ to one set of servers and /api/ to another, route based on the visitor’s language, or terminate HTTPS for the whole pool. It’s more flexible but does a little more work per request. For a website, layer 7 is usually what you want.
There’s also a difference in where the balancer runs. A software load balancer is a program running on an ordinary server — flexible, cheap, and what most projects use. A hardware load balancer is a dedicated physical appliance built for raw speed, found mostly in large enterprises. And in the cloud, the provider offers load balancing as a managed service you switch on without running anything yourself, which is how a lot of modern setups handle it. If the difference between running your own box and renting managed infrastructure is fuzzy, our overview of shared vs VPS vs dedicated vs cloud puts those options in context.
The balancer itself can become the single point of failure
There’s an irony worth naming: if all your traffic flows through one load balancer and that machine dies, you’re down again — you’ve just moved the single point of failure rather than removing it. Serious setups run the balancer itself redundantly (two or more, with automatic failover between them) so there’s no single box whose death takes everything offline. It’s the kind of detail that’s easy to forget until the day it bites.
A mental model that sticks
If you remember nothing else, hold onto this: a load balancer is a traffic director standing in front of a team of interchangeable workers. Its three core duties are to spread the work fairly, to notice when a worker is down and route around it, and to keep the team looking like a single, dependable service from the outside.
Everything else — the algorithms, sticky sessions, layer 4 versus layer 7 — is detail layered on top of that one idea. Get the core picture firmly in place, and the rest slots in naturally as you need it. And because the load balancer is itself a kind of server receiving requests on behalf of others, it helps to be solid on what a web server is before going deeper here.
Wrapping up
Let’s gather it all in one place:
- Load balancing spreads incoming requests across several identical backend servers so no single one is overwhelmed. The load balancer is the public-facing piece that decides who answers each request.
- It solves two problems at once: capacity (scale out by adding more cheap servers instead of one giant one) and reliability (one server dying no longer takes the whole site down).
- Servers are picked using an algorithm — round robin (take turns), least connections (send to the least busy), weighted versions for unequal hardware, or IP hash for stickiness.
- Spreading users across servers breaks per-server memory, causing the sticky session problem. Fix it with session affinity, or better, make servers stateless with a shared session store.
- Health checks let the balancer detect dead backends and route around them, which is what actually delivers the “stays up” promise.
- Balancing can happen at layer 4 (fast, connection-level) or layer 7 (smart, HTTP-aware), and run as software, hardware, or a managed cloud service. Don’t forget to make the balancer itself redundant.
Once you’re comfortable with how traffic gets spread across many machines, the natural next step is seeing how a single server hosts several different sites at once — which is exactly what virtual hosts are for.
