Picture three identical web servers, all ready and willing to answer. A visitor knocks. Someone has to decide: which one takes this request? That little decision, made thousands of times a second, is the whole job of a load balancer. The obvious move is to grab an external proxy off the shelf — another daemon to run, another config language to learn, another moving part to wake you up at night. We did the less obvious thing instead. We built the load balancer into the runtime itself, so that ocifbsd proxy feels like as natural a part of the toolbox as run or build. It’s not a bolt-on. It came in the box.
Layer 4, and proud of it
Here’s a design choice worth dwelling on, because it’s deliberate and it’s a little contrarian. The proxy is protocol-agnostic: it balances TCP connections, not HTTP requests. It works at layer 4, down where the plumbing is, rather than layer 7, up where the conversation happens. Why on earth would you give up all that HTTP-awareness?
Because a layer-4 balancer doesn’t care what the bytes flowing through it actually are. HTTP? Sure. A database wire protocol? Fine. Some bespoke thing you invented last Tuesday? It’ll move that too, without judgment. It shuttles connections to backends and then gets politely out of the way. That makes it simpler, faster, and dramatically more reusable than a proxy that has to understand and parse every protocol it carries. Right now it’s spreading web traffic across the cluster nodes — but it would sit just as happily in front of almost anything else you could name.
Four ways to pick a winner
“Fair” turns out to mean different things for different workloads, so how the proxy chooses a backend is configurable. There are four flavors, and each has its moment:
- Round-robin is the default, and it’s exactly as honest as it sounds — it simply takes turns, one after another, no favorites.
- Random scatters load statistically and keeps no shared state to do it, which makes it beautifully cheap.
- Least-connections sends each new arrival to whichever backend is currently doing the least work — a lifesaver when some requests are quick and others drag on forever.
- Source-hash maps a given client to the same backend every time, which is how you get sticky sessions when you actually want them.
And underneath all four, there’s a safety net: a dead backend gets noticed and quietly failed over to the next one in line. One sick node doesn’t get to drag your visitors down with it. That’s the difference between a load balancer and a single point of failure with extra steps.
Built to use the whole machine — and one bug it took to get there
A single accept loop is a traffic jam waiting to happen — one poor thread trying to greet everybody at once. So the proxy pre-forks a whole pool of workers, one per CPU core, all sharing a single listening socket with a nice deep backlog, each one dialing out to backends on its own. That’s the arrangement that let it stay calm and healthy while the stress test poured thousands of concurrent connections through it. Every core pulling its weight.
But I won’t pretend it worked perfectly on the first try, because that would be a boring and dishonest story. An early version had a nasty little bug: it tore down both halves of a connection the instant either side signalled it was done talking. That sounds reasonable until you meet a client that half-closes — finishes sending, but is still waiting to receive — at which point the proxy would guillotine the response mid-sentence and hand back a truncated page. The fix was proper half-close handling: shut down only the direction that’s actually finished, and keep patiently draining the other until it’s done too. That one change turned a scatter of intermittent errors into a clean, straight line. And honestly, that’s the quiet argument for building your own: a load balancer you own is a load balancer you can actually fix.