Blog

For a while, this “cluster” had a dirty secret. The web tier was genuinely three nodes — three copies of WordPress and Nginx, load-balanced, any of which could serve you. But behind them sat a single MariaDB and a single Redis, both on one node. Lose that node and the whole thing goes dark. A cluster with one heart isn’t a cluster; it’s a single point of failure wearing a crowd for a costume. This is the story of fixing that — making the database and the cache into real, synchronous clusters where any node can be written to and every node has the same data.

The goal: three that act as one

We wanted a MariaDB Galera cluster — three database instances kept in lock-step by synchronous replication, so a write committed on any node is instantly present on all three — and a 3-master Redis Cluster for sessions and cache, where the keyspace is shared across all three and a client reaching any node finds the same data. Both images already had what they needed: MariaDB shipped with the Galera provider, Redis 8 speaks cluster natively. It should have been a config exercise. It was not.

The wall: you cannot cluster across NAT

The first attempt to add a Galera node failed instantly with a cryptic message: Will never receive state. Need to abort. The cause was the network. Each node’s containers lived on the same private subnet (10.88.0.0/24), isolated behind that node and reachable only through port-forwards on the node’s public IP. Galera’s state transfer (SST) needs a joining node to advertise an address the donor can connect back to — but the container could only bind its private IP, not the public one it had to advertise. That mismatch is fatal, and it’s a well-known wall: you can’t run a peer-to-peer cluster protocol across one-to-one NAT.

The fix: give every container a real address

The clean solution was to stop hiding the containers behind NAT and make them directly routable. We renumbered each node’s pod network onto its own distinct subnet — 10.88.1.0/24, 10.88.2.0/24, 10.88.3.0/24 — turned on IP forwarding, and added routes between the nodes so a container on one node can reach a container on another by its own address, with no NAT in the middle. Suddenly every database and cache instance had a real, reachable identity. It touched the whole web tier’s wiring too, and it was worth every minute.

With that in place, Galera formed on the first try. We converted the existing database in place — bootstrapping it as the first cluster member without losing a single row — then brought up the other two, which pulled a full copy over the now-routable network and joined. Three members, status Primary, fully synced.

Proof, not promises

A cluster you haven’t tested is a rumor. So we wrote a row directly into the database on node 2 and immediately read it back from nodes 1 and 3 — there it was, identical, on both. We set a key in Redis on node 2 and fetched it from node 1 and node 3 — the cluster redirected each request to the right shard and returned the same value. Then we pointed each node’s WordPress at its own local database and the shared Redis Cluster. Every node now reads and writes locally, and the data is everywhere at once.

Born from one file

The last piece was making all of this reproducible. The entire cluster — the routed networks, the Galera and Redis instances, the web tier on every node, the load balancer in front — is now described in a single Ensemble manifest, and ocifbsd stack up -f deploys it across the nodes. We proved it by wiping a node’s containers and running that one command: they came back, in the right place, from the file. A cluster should be able to describe itself and rebuild itself. Now this one can — and there isn’t a single heart left to stop it.

We already told the story of pushing this cluster until it bent. This is the follow-up with the receipts: a fresh, methodical stress test where we turned the concurrency dial up in stages, wrote down every number, and — most importantly — figured out which part of the stack gives way first. The short version: the load balancer we built has enormous headroom, and the two real limits are exactly the ones you’d predict once you see them.

How the test was run

For the duration of the test we flipped Cloudflare’s proxy off, so the load hit our own origin directly instead of being absorbed by a CDN — then flipped it back on when we were done. Then we drove traffic with hey at rising concurrency levels, against two very different kinds of request: the cached front page (served straight from Nginx’s FastCGI cache) and an uncacheable page that has to wake up PHP-FPM and query MariaDB on every hit. Those two numbers tell completely different stories, which is the whole point.

Finding 1 — the app tier: ~30 renders a second, then 502s

Uncacheable requests are the honest test of the application, because every single one travels the full path: Nginx → PHP-FPM → MariaDB → back again. Here’s what happened as we raised concurrency (each level held for 12 seconds, fresh connection per request):

concurrency   throughput   slowest request   result
    50          31 rps         3.2 s          all 200 OK
   100          31 rps         4.8 s          all 200 OK
   200          29 rps         8.8 s          all 200 OK
   400          27 rps        17.4 s          all 200 OK  (deep queue)
   800         229 rps        26.0 s          933 x 200  +  7093 x 502  <-- break

The application tier holds a steady ~30 uncacheable renders per second across the three nodes. Notice throughput barely moves from 50 to 400 concurrent — it's saturated the whole time — while the latency climbs and climbs as requests pile into the queue: three seconds, then five, then nine, then seventeen. At 800 concurrent the queue finally overflows and the upstream starts refusing work: 88% of requests come back as 502 Bad Gateway. That's the break. It isn't mysterious — it's a finite pool of PHP-FPM workers and one shared database, doing exactly as much as they can and no more.

Finding 2 — the cache: 30× the throughput, until the edge gives

The cached front page is a different universe. With reused connections, the public site served this:

concurrency   throughput   average latency   result
   200         861 rps        0.17 s          all 200 OK
  1000          54 rps       16.5 s           all 200, but crawling
  3000         100 rps       18.1 s           all 200, but crawling

861 requests a second at moderate concurrency — roughly thirty times the dynamic ceiling, which is the entire reason the FastCGI cache exists. But look what happens past a thousand concurrent connections: throughput falls off a cliff and latency balloons to sixteen seconds, even though serving a cached page costs almost nothing. When near-free work grinds to a halt, the bottleneck isn't the work — it's the connections themselves. Something in front of the cluster was choking on the sheer number of simultaneous sockets. So we went looking for it.

Finding 3 — the load balancer is not the bottleneck

To find the real ceiling we cut out the public path entirely and pointed the load straight at the ocifbsd proxy on the internal network — no TLS, no home-internet gateway in the way. Same cached request, same tool, same machine generating load:

concurrency   throughput   average latency   result
    50        1308 rps        0.04 s          all 200 OK
   200         892 rps        0.21 s          all 200 OK
  1000        1012 rps        0.54 s          all 200 OK   <-- public path did 54 here
  3000         462 rps        1.6 s           all 200 OK
  6000         234 rps        9.8 s           all 200 OK, zero errors

There's the answer. Hit directly, the proxy does over a thousand requests a second at a thousand concurrent connections — right where the public path had collapsed to fifty-four — and it keeps returning correct responses with zero errors all the way up to six thousand concurrent. The native ocifbsd load balancer has headroom to spare. The high-concurrency collapse we saw from the outside lives above it: the TLS-terminating ingress and the consumer-grade gateway's NAT table, which is simply not built to juggle thousands of simultaneous connections. In production that layer is exactly what a CDN like Cloudflare is for — which is why it sits in front.

What actually breaks, and what to do about it

• Dynamic content saturates at ~30 req/s and hard-fails near 800 concurrent. This is the shared PHP-FPM + single MariaDB. The fix is horizontal: more web replicas raise the worker count, and eventually the database tier has to scale out too. More RAM and CPU raise the number; they don't change the shape.
• Cached content does ~860 req/s through the front door and would do far more if the front door were bigger. The cache is doing its job — the ceiling here is the edge, not the app.
• The ocifbsd proxy sustains 1000+ req/s at 1000 concurrent and never errors up to 6000. The piece we built ourselves is the piece with the most room. That's a good place to be.
• Throughout all of it, when a request failed it failed cleanly — a 502, not corrupted content — and the circuit-breaker and failover kept the healthy paths fast. A system that degrades honestly under overload is one you can actually reason about.

None of these limits is a surprise, and that's the best thing we can say about them. We can point at the exact request rate where each tier tops out, explain why, and name the knob that moves it. That's what a stress test is for — not to prove nothing breaks, but to know precisely what breaks, when, and what to do next.

Here is the question every “highly available” system should have to answer out loud: what actually happens when a machine dies? Not in a diagram — in reality, with real traffic flowing, when you walk over and pull the plug. We did exactly that to this cluster, and the story of what broke, what held, and what healed itself is the most honest thing on this whole site.

Pulling the plug (the right way)

The first lesson came before the test even started. The tempting way to fake a node failure is to stop the containers on it — but that turns out to be a trap, because ocifbsd’s lifecycle can’t restart a stopped container in place, and cleaning up drags you into a fragile rebuild. The honest way is also the simpler one: power the whole virtual machine off. So that’s what we did — vm stop on one of the three nodes, a genuine “the server just died” event, reversible with a single vm start when we were done.

What held: traffic failover

With a third of the cluster suddenly gone, the site didn’t so much as flinch. Every request still came back 200, served by the two surviving nodes. The native ocifbsd load balancer noticed the dead backend, skipped it, and sent the connection to a replica that was alive. That’s failover, and it’s the baseline you’d hope for. But hoping isn’t knowing, and when we looked closely at the timing, we found something worth fixing.

The slow bleed, and the circuit-breaker

A dead machine doesn’t politely refuse your connection — it just goes silent. So every time the balancer tried the downed node, it had to wait out a two-second timeout before giving up and trying a live one. The request still succeeded, but it took two seconds instead of forty milliseconds. And because the balancer rotates through backends evenly, roughly one request in three kept paying that toll, over and over, for as long as the node stayed down. Correct, but bleeding.

The fix is an idea borrowed from electrical panels: a circuit-breaker. Give every backend a little shared health record. Each time a connection to it fails, count it; after three failures in a row, trip the breaker — mark that backend dead and stop even trying it for a ten-second cooldown. The measurement tells the whole story. Here’s the latency of twenty requests, in seconds, right after the node went down:

0.036  0.001  2.04   0.035  0.041  2.04   0.001  0.001  2.23   0.043 ...
                └──────── three detections ────────┘
                                                    ▲ breaker trips here
after: 0.043  0.001  0.042  0.040  0.001  0.039  0.036  0.043 ...  (all fast)

Three requests pay the two-second detection cost. Then the breaker trips, the dead node is short-circuited entirely, and every request afterward is fast again — while still returning correct content, because the traffic simply flows to the healthy replicas. The cost of a dead node dropped from “forever, on a third of requests” to “three requests, once.”

What didn’t heal: coming back from the dead

Then we powered the node back on, expecting it to slot back into the cluster. It didn’t. The virtual machine booted fine — but its containers stayed dark. Nothing on the node was responsible for starting them at boot, so the web server and app that had been running were simply… not. The machine was alive; the work it was supposed to be doing was not. That’s the difference between a server rebooting and a service recovering, and we’d been missing the second half.

The restart policy that was missing

So we built it: a small supervisor that gives every container a restart policy. It reads a manifest of what’s supposed to be running — each container’s image, its network address, and a policy of always, on-failure, or no — and a few times a minute it makes reality match. If a container that should be running isn’t, it recreates and restarts it, reapplying the network address it’s supposed to have. A crashed jail comes back within seconds. A rebooted node brings its whole workload back on its own, including re-establishing the pod network’s gateway so the node can talk to its containers again.

Crucially, it’s careful: a container that’s already healthy is left completely alone. We confirmed that by deploying the supervisor onto a node whose containers were happily running — same containers, same identities, nothing disturbed. It only acts when something is actually wrong.

The full loop

Put the pieces together and the cluster now survives a node death end to end, on its own. A machine dies: the circuit-breaker trips within three requests and short-circuits it, so visitors never feel it. The machine comes back: the supervisor restarts its containers and repairs its networking, and the load balancer’s breaker — which after its cooldown allows a single probe through — finds the node answering again and quietly folds it back into rotation. No pager, no human, no downtime the visitor could measure. We know because we watched it happen, plug and all.

There’s a real difference — wider than most people expect — between software that passes its tests and software that has actually been used. The first kind is clever. The second kind is wise, and it earned that wisdom the hard way. ocifbsd got the second kind of education, because early on we made a decision that turned out to matter more than almost any other: we made it run something real, something unforgiving, something that would embarrass us in public if it broke. Namely, the very website you’re reading. Dogfooding — running your own tool in earnest, as a customer rather than an author — is far and away the fastest way to find the gap between “works in the demo” and plain old “works.”

A real app asks the hard questions

WordPress is not a toy workload, and that’s exactly why it was the right choice. It wants a web server, a PHP runtime, a database, and a cache — all networked together, and all quietly assuming they’re running on a perfectly normal machine that behaves the way machines have always behaved. Standing that whole stack up on ocifbsd immediately surfaced a stack of things a cheerful little hello-world container would never in a million years reveal.

Services that needed devfs present, and sulked without it. A database that was only reachable if the network overlay got applied at exactly the right moment — not a moment later. A PHP process that simply assumed a working loopback interface and fell over when it didn’t find one. And here’s the crucial bit: every single one of those became a fix in the runtime rather than a grubby workaround buried in the app’s config. That is the entire point of dogfooding, distilled. The pain lands somewhere you can fix it permanently, for everyone, instead of somewhere you have to keep apologizing for.

Features that exist only because the site demanded them

Some of ocifbsd’s most genuinely useful capabilities exist for one reason and one reason only: this deployment stood up, crossed its arms, and demanded them. The stats command was born straight out of wanting the site to display its own live resource usage — and now that same JSON feed drives the Machine Room dashboard and the cluster map you can go poke at right now. The proxy grew its algorithms, its sticky sessions, and its multi-worker accept pool under the real, sweaty pressure of actually balancing cluster traffic, not in a vacuum where we guessed at what might someday be nice. The networking fixes came from two containers that genuinely, urgently needed to talk to each other and couldn’t.

None of it was speculative. None of it was a feature added because a roadmap said so or because it would look good on a slide. Every bit of it was pulled into existence by a running system that wanted it — which, if you ask me, is the best possible reason for any feature to exist. Necessity is a far better product manager than imagination.

The site as a test that never clocks out

And here’s the part I like best of all: the dogfooding never stops. This site stays up. Which means ocifbsd is under continuous, honest, unblinking evaluation — not a test suite that runs for ninety seconds and then exits satisfied, but a living workload that has to keep serving correct pages, keep migrating sessions between nodes, keep reporting accurate stats, day after day after day, whether or not anyone’s watching.

When something regresses, a real website breaks — visibly, publicly, right now — and that is a feedback loop no amount of unit testing on Earth can replace. There’s no hiding from a broken homepage. The runtime got sharper precisely because it always had something to prove, out in the open, all the time, with the stakes real. That, in the end, is what “forged by dogfooding” actually means. Not a slogan — a standing dare.

Writing the code is just the first draft. I know that’s a slightly deflating thing to hear if you’ve just spent a week getting something to compile and run, but it’s true, and it’s especially true for anything that’s going to face a hostile network. What actually turns a runtime into something you’d trust with a public-facing service is everything that happens after that first draft: the review. ocifbsd went through more than twenty distinct rounds of adversarial review — twenty-two, to be exact — and every single one of them was given permission to be ruthless. The commit history keeps them all, a numbered parade of “review batch” fixes, and honestly the list reads like a compact education in how systems software really breaks.

The bugs that lurk in C

If there was a recurring villain across these rounds, it was memory. Several batches went hunting for one particularly subtle and dangerous pattern: reallocating a buffer straight back into the same pointer. It looks harmless — it looks like the obvious thing to write — but it leaks the old allocation if the realloc fails, and it can leave you clutching a dangling reference when it succeeds. A dedicated growth helper was introduced to do it correctly, and then the bad pattern was rooted out everywhere it had taken hold: the list builders, the registry client, the runtime core.

Other batches went after equally nasty company: a use-after-free in the log daemon’s ring buffer, a cross-thread use-after-free that surfaced while listing cluster nodes, and a ring buffer cheerfully calling munmap on memory that had actually been handed out by calloc — a mismatch that’s invisible right up until it isn’t. These are exactly the kind of failures that sail through every functional test with a smile and then crash your service at 3 a.m. under real load, which is the worst possible time to meet them. Better to meet them in review.

Security as a first-class citizen, not an afterthought

A container runtime handles untrusted input as a basic condition of its existence — images, registry data, API calls, all of it arriving from who-knows-where with who-knows-what intentions. So security review earned its own dedicated series of batches, and the list of what got closed is worth reading slowly. Command injection through popen in the certificate and metrics paths was eliminated. Path traversal in the orchestration and networking code was guarded against. Decompression was capped so that a hostile image couldn’t unfold into a decompression bomb and eat the host alive.

And it kept going. Token authentication was bound to the authenticating identity to slam a bypass shut. JWT handling was fixed to close an out-of-bounds read. Cluster peer identity was pinned rather than trusting a CA chain alone — because in a cluster, “who are you, really?” is a question you want a very firm answer to. Key files were created 0600 from the very first moment, never lying around readable even briefly. And untrusted strings headed into hand-built JSON were properly escaped, to shut down log and audit injection before it could start. None of these are glamorous. All of them are the difference between a toy and a tool.

Why twenty-two rounds, and not just one?

You might reasonably wonder why it takes twenty-two passes to review one codebase. Couldn’t a sufficiently careful person just… do it once, properly? And the answer is genuinely interesting: no, because real review converges, it doesn’t finish. Each round fixes a whole class of problem — and the act of fixing one class quietly makes the next class visible. Tidy up the memory management, and suddenly the concurrency bugs step out of the shadows they’d been hiding in. Lock down the obvious injection points, and the subtler identity bugs finally stand out against the cleaner background.

So twenty-two batches was never a sign the code was bad. It was the mechanism by which the code became good — each pass sharpening the lens for the next. And it’s worth saying plainly: every shiny feature in the rest of these posts is only trustworthy because this unglamorous, repetitive, deliberately adversarial work happened first, out of sight, before any of it went live. The fun stuff rests on the boring stuff. It always does.

ocifbsd did not descend from the clouds fully formed. Nothing worth using ever does. It was built the way real software is always built — through a great many small, deliberate commits, each one fixing a single thing or adding a single capability, stacked patiently on top of each other over months, until one day there was a working runtime where before there had only been an idea and a hunch. Three months. Three hundred and ninety-five commits. Roughly a quarter-million lines of C. The repository is the honest record of all of it, and I’d gently suggest you read it as a story rather than a changelog. It reads better that way.

A quarter-million lines, and why they’re in C

Let’s address the language question up front, because someone always asks. The runtime is written in C — not out of nostalgia, not to prove a point, but because C is the right altitude for a tool whose entire job is to speak fluently and directly to the FreeBSD kernel. Jails, VNET, RACCT, pf, devfs — these are the primitives ocifbsd drives, and they’re happiest when you talk to them in their own language, without a tall stack of abstractions muffling the conversation. When your whole purpose is to be the thin, sharp layer between a container and the kernel, you want to stand as close to that kernel as you reasonably can.

The tree spans a satisfying range of subsystems: image handling, an OCI-to-jail translation layer, a networking configuration model, a registry client, an orchestration state machine, a logging daemon, a security daemon, and the user-facing command surface that ties it all together. And here’s the design instinct I most admire in it — each of those subsystems is small enough that one person can hold it in their head, and they’re wired to each other through narrow, legible interfaces rather than a tangle of shared everything. Small pieces, clearly joined. It’s the difference between a machine you can service and a machine you can only replace.

The features you can point at

Some commits are landmarks — the ones where you can point at the history and say, “there, that’s the day the tool could do a new thing.” build arrived, and suddenly the platform could assemble its own images from a Containerfile. stats arrived to report per-container resource usage as JSON — and that very feed is what powers the live dashboard on this site, the numbers you can go watch move right now. proxy arrived as a native layer-4 load balancer, then kept growing: load-balancing algorithms, sticky sessions, correct half-close handling, a pre-forked multi-worker accept pool. And networking got a whole run of fixes that transformed VNET from “attached to the bridge but completely mute” into containers that actually route packets like grown-ups. The lovely thing is you can trace every one of these as a clean sequence of commits, each with a message that says plainly what changed and, more importantly, why.

The unglamorous discipline that holds it all up

But here’s the part that never makes it onto a feature list, the work nobody throws a launch party for — and it’s the part I want to end on, because it’s what actually makes the rest trustworthy. Indentation was converted to hard tabs to match FreeBSD’s style(9). Trailing whitespace was stripped across the entire tree. Ignored return values were explicitly marked as intentional, so the next reader knows it was a choice and not an oversight. A clunky fork+ifconfig was replaced with a direct if_nametoindex(3) call. Quadratic string-building was rewritten to run in linear time using open_memstream.

None of that changes what the software does. Not one bit of it shows up in a demo. But all of it changes whether the next person — possibly you, possibly future-me at 2 a.m. — can safely reach into the code and change it without holding their breath. Three months of that quiet, repetitive, unglamorous discipline is the real reason the interesting features could be built on solid ground instead of on sand. The flashy commits get the applause; the housekeeping is what keeps the building standing.

Every confident claim on this site — that the cluster is real, that your session follows you around, that a load balancer written from scratch in FreeBSD can hold a genuine crowd — is worth exactly nothing until something angry tries to knock it flat. Talk is free. Load is not. So we stopped talking and started swinging. What follows is the log of the day we pointed a steadily growing swarm of concurrent visitors at the three-node cluster and watched, patiently, for the first crack to appear.

The rig on the table

No mockups, no simulators, no comfortable lies. The target was the clustered deployment exactly as you’re reading it right now: three FreeBSD nodes — fb16-1, fb16-2, fb16-3 — each running an identical image of Nginx (with a FastCGI cache out front) sitting ahead of PHP-FPM and WordPress, all wrapped in native OCI jails managed by ocifbsd. Standing guard in front of the whole thing is the ocifbsd proxy: that protocol-agnostic layer-4 balancer baked right into the runtime, spreading connections round-robin and quietly failing dead backends over to the next. Behind the nodes, one shared Redis holds every visitor’s session, and one shared MariaDB is the single source of truth. The load hit the same TLS front door, the same proxy, and the same jails that serve real traffic on any ordinary day. Nothing was staged.

Phase one: 125 at once, and the cache earns its paycheck

We opened gently — if you can call 125 simultaneous clients hammering the front page and the inner pages “gentle.” And crucially, every one of those clients wasn’t just checking that a response came back. It was checking that the bytes it got were the correct bytes: not a truncated page, not an error dressed up in a 200’s clothing, not garbage. The verdict: 100% correct content. No wrong pages, no corruption, not one byte out of place. That’s the boring result you desperately want.

But the number that made everyone lean in was latency — and this is where the FastCGI cache stepped up and did something dramatic. A cold WordPress render is a lot of work: PHP boots, MariaDB gets interrogated, the page is assembled brick by brick. Under contention, that cold render clocked in around 5.3 seconds. Painful. But with the cache keyed per-URL, configured to skip only on cookies, POSTs, and wp-admin, a warm hit came screaming back in roughly 63 milliseconds. That is not a typo, and I checked it twice myself. Caching the right things chopped the served latency by around eighty-fold. It’s an old lesson, but it’s worth relearning every single time it happens: the fastest database query in the world is the one you never had to make.

The session that flatly refused to get lost

A balancer that flings you to a new machine on every click is worse than useless if your session evaporates the moment it does. So right in the middle of the test, we kept an eye on one single session’s counter while the proxy deliberately knocked it around the cluster like a pinball. It climbed: 1 → 2 → 3 → 4 → 5 → 6. And every one of those increments was served by a different node, under the same session id. Because the session lives in shared Redis rather than on any one node’s local disk, it just quietly followed the visitor from fb16-1 to fb16-2 to fb16-3 and back around again, never missing. That’s the thing that makes the web tier genuinely stateless — and stateless is exactly the property that makes it scale sideways.

Phase two: crank it until something bends

Then we stopped being polite. We turned the dial up and up — past 250, past 500, past 1,000 — and finally parked an outright firehose of 2,000 concurrent connections on the cluster to go find the edge of the map. And — good news for the honesty of this post — we found it.

The genuinely interesting part wasn’t that it bent. Everything bends eventually. It was where. The proxy, notably, did not fall over — after one real fix (I’ll get to it) it just kept calmly accepting and distributing connections across every core, unbothered. The ceiling turned out to be the application tier: a finite pool of PHP-FPM workers, and behind them, that single shared MariaDB. The uncacheable requests — logins, POSTs, admin actions — can’t be short-circuited by the cache; they have to reach all the way to PHP and the database. And past a certain point, there simply aren’t enough workers or database connections to go around. At the 2,000 extreme, roughly nine in ten uncacheable requests were turned away or slowed rather than served instantly. That’s not the proxy failing. That’s the proxy being scrupulously honest about a backend that’s run out of hands.

And it’s worth being precise about which knob does what, because this is where a lot of intuition goes wrong. Throwing more RAM and CPU at the nodes raises the capacity — more workers, more cache headroom, a bigger number before things saturate. Genuinely helpful! But it does not change the shape of the limit. A single database of record and a fixed worker pool will always, always have a saturation point somewhere; you’re just moving it. The honest way to push the ceiling further up is more replicas, and eventually a database tier that scales out too. Knowing which knob addresses which limit — that’s the entire reason you run a stress test instead of guessing.

Two bugs the crowd shook loose

Here’s a thing they don’t tell you: a big enough crowd is a debugging tool. Two bugs that had been hiding in plain sight came tumbling out under the pressure.

• The half-closed connection. Under heavy load the proxy started handing back the occasional truncated response — maddening, because it was intermittent. The culprit was the pump loop tearing down both directions of a connection the moment either side signalled EOF. So when a client half-closed its upload while the response was still streaming back, the response got its sentence cut off mid-word. The fix was proper half-close handling: shut down only the finished direction with SHUT_WR, and keep draining the other until it, too, is genuinely done. After that, the 5xx blips under load simply vanished.
• The node that couldn’t reach itself. One replica’s WordPress kept throwing 500s, and the reason was delightfully sneaky: it was configured to reach the database at its own node’s public IP — and the packet-filter redirect that maps that public IP down to the database container does not apply to traffic a node originates to itself. The fix was to point that replica straight at the database’s container address on the pod network, sidestepping the redirect entirely. A small, very FreeBSD-flavored lesson about exactly where pf rules do and do not fire.

What the whole exercise actually proved

Three things, concretely, and I’ll stand behind each one. First: the cluster serves correct content under real concurrency — not “approximately correct,” not “correct-ish,” but every byte. Second: sessions migrate, which means the web tier is truly stateless and any node can answer any request. Third — and this is the one I’m proudest of — the failure mode is understood and, frankly, boring. The app tier saturates predictably, the proxy stays honest about it, and the path to more headroom (more replicas, a scale-out data tier) is clearly signposted rather than shrouded in mystery. A system you can push to its absolute limit on purpose, and then calmly explain afterward, is a system you can actually trust in production. That’s the whole reason we turned the dial all the way to 2,000.

Here’s a trick question that quietly decides whether a cluster is real or just for show. If your load balancer sends you to a different machine on every single click, what happens to your session — your login, your cart, your half-filled form? If the answer is “you get logged out,” then I’m sorry to tell you that you don’t actually have a cluster. You have three separate websites standing on each other’s shoulders wearing a long trench coat, hoping nobody notices. Making a session follow the visitor is the single thing that turns three nodes into one system.

The quiet problem with local state

By default, PHP does something that feels perfectly innocent: it stores each visitor’s session on the local disk of whichever server happened to handle the request. On a single machine, that’s flawless. It’s fast, it’s simple, it just works. But spread across many machines it becomes quietly fatal. Node A took your login and wrote it down. Nodes B and C? They’ve genuinely never heard of you. You’re a stranger to two-thirds of your own cluster.

And this is the deeper lesson, the one worth carrying to other systems entirely: as long as any part of handling a request depends on state that lives on one specific node, the load balancer’s beautiful freedom to send you anywhere stops being a feature and turns into a liability. Flexibility upstream demands statelessness downstream. You can’t have one without the other.

Get the state off the node entirely

So the fix isn’t to make the nodes share their disks or gossip about who logged in where. The fix is cleaner and a little bolder: stop storing session state on any node at all. Every replica gets pointed at one shared Redis, and PHP is told to keep its sessions there instead of on local disk. Now your session doesn’t belong to node A, or B, or C. It belongs to the cluster. Whichever replica the balancer hands your next request to, that replica looks your session up in the same shared Redis and finds precisely what the previous one wrote a moment ago.

The web tier becomes stateless — and statelessness is the magic property, the one that quietly makes horizontal scaling possible in the first place. When no replica is holding anything precious that the others lack, adding capacity stops being an anxious surgery and becomes trivial: you just add another interchangeable replica and walk away. They’re all the same. None of them is special. That’s the whole point.

Watching it actually happen

Now, I could just draw you a diagram and swear it’s true. But this isn’t a hope we sketched on a whiteboard — it’s a behavior we sat and watched with our own eyes. During testing we took a single session and incremented its counter across a run of requests, while the proxy was deliberately, mischievously bouncing that session from node to node between each one. And the counter climbed, clean as anything: 1, 2, 3, 4, 5, 6. Every step of the way. Except here’s the kicker — each of those increments was served by a different node, under the same session id. The state never skipped a beat.

The visitor, if there had been a real one, would never have felt a thing. They’d never have known they’d been quietly shuffled between three physical machines mid-visit — and that obliviousness is exactly the goal. Session migration is invisible when it works, and its invisibility is the feature. You roam freely across the cluster, and your session simply… comes along for the ride, like it was never anywhere else.

Picture three identical web servers, all ready and willing to answer. A visitor knocks. Someone has to decide: which one takes this request? That little decision, made thousands of times a second, is the whole job of a load balancer. The obvious move is to grab an external proxy off the shelf — another daemon to run, another config language to learn, another moving part to wake you up at night. We did the less obvious thing instead. We built the load balancer into the runtime itself, so that ocifbsd proxy feels like as natural a part of the toolbox as run or build. It’s not a bolt-on. It came in the box.

Layer 4, and proud of it

Here’s a design choice worth dwelling on, because it’s deliberate and it’s a little contrarian. The proxy is protocol-agnostic: it balances TCP connections, not HTTP requests. It works at layer 4, down where the plumbing is, rather than layer 7, up where the conversation happens. Why on earth would you give up all that HTTP-awareness?

Because a layer-4 balancer doesn’t care what the bytes flowing through it actually are. HTTP? Sure. A database wire protocol? Fine. Some bespoke thing you invented last Tuesday? It’ll move that too, without judgment. It shuttles connections to backends and then gets politely out of the way. That makes it simpler, faster, and dramatically more reusable than a proxy that has to understand and parse every protocol it carries. Right now it’s spreading web traffic across the cluster nodes — but it would sit just as happily in front of almost anything else you could name.

Four ways to pick a winner

“Fair” turns out to mean different things for different workloads, so how the proxy chooses a backend is configurable. There are four flavors, and each has its moment:

• Round-robin is the default, and it’s exactly as honest as it sounds — it simply takes turns, one after another, no favorites.
• Random scatters load statistically and keeps no shared state to do it, which makes it beautifully cheap.
• Least-connections sends each new arrival to whichever backend is currently doing the least work — a lifesaver when some requests are quick and others drag on forever.
• Source-hash maps a given client to the same backend every time, which is how you get sticky sessions when you actually want them.

And underneath all four, there’s a safety net: a dead backend gets noticed and quietly failed over to the next one in line. One sick node doesn’t get to drag your visitors down with it. That’s the difference between a load balancer and a single point of failure with extra steps.

Built to use the whole machine — and one bug it took to get there

A single accept loop is a traffic jam waiting to happen — one poor thread trying to greet everybody at once. So the proxy pre-forks a whole pool of workers, one per CPU core, all sharing a single listening socket with a nice deep backlog, each one dialing out to backends on its own. That’s the arrangement that let it stay calm and healthy while the stress test poured thousands of concurrent connections through it. Every core pulling its weight.

But I won’t pretend it worked perfectly on the first try, because that would be a boring and dishonest story. An early version had a nasty little bug: it tore down both halves of a connection the instant either side signalled it was done talking. That sounds reasonable until you meet a client that half-closes — finishes sending, but is still waiting to receive — at which point the proxy would guillotine the response mid-sentence and hand back a truncated page. The fix was proper half-close handling: shut down only the direction that’s actually finished, and keep patiently draining the other until it’s done too. That one change turned a scatter of intermittent errors into a clean, straight line. And honestly, that’s the quiet argument for building your own: a load balancer you own is a load balancer you can actually fix.

One container is a peaceful thing. It sits there, it does its job, and nobody has to think very hard about it. But the moment you have two containers that need to talk to each other — a web server that needs a database, the oldest pairing in the book — something sneaky happens. You are quietly, without signing up for it, no longer in the container business. You’re in the networking business. And getting containers on FreeBSD to reach each other, reach out to the wider internet, and be reachable from the outside world was the least glamorous, most consequential work behind this entire site.

VNET: giving every container its own little internet

The bedrock of all of this is VNET, FreeBSD’s network-stack virtualization. It’s a wonderful piece of engineering, and the idea is easy to fall in love with. Instead of every container elbowing for room on the host’s shared interfaces, each one gets its own stack: its own epair (think of it as a virtual patch cable with two ends), its own routing table, its own private view of the network world. A container isn’t a process squatting on the host’s address anymore. It’s a real network citizen with its own front door.

ocifbsd takes all those virtual interfaces and plugs them into a shared bridge — the pod network — so that containers living on the same node all sit on one friendly little subnet (here it’s 10.88.0.0/24) and can call each other by address, directly, no fuss. This is why VNET is non-negotiable for this project. It’s the difference between containers that merely coexist and containers that can genuinely cooperate.

The bugs that live in the seams

Now, if you’ve done any networking, you already know where the bugs hide. Never in the middle of things — always in the handoffs, the seams, the little moments where one component passes responsibility to another. This project was a textbook case, and I want to walk you through it because the bugs are genuinely instructive.

First, containers were coming up dutifully attached to the bridge — but without a working default route. So packets would cheerfully make it onto the subnet and then just… stop, like a traveler who reached the airport but never got a boarding pass. The fix was to actually apply the configured gateway when the interface comes up. Second, the network configuration was being written down correctly but not overlaid at container start, so a fresh run would boot up wearing yesterday’s settings; re-applying the netcfg overlay at launch sorted that out. And then even humble loopback got its say: services that expected to find themselves at 127.0.0.1 inside their own container needed the loopback interface configured too, not just the fancy external one. Every single one of these fixes was only a few lines of code. Every single one of them cost an afternoon to find. That’s networking for you.

From the little subnet out to the whole world

Getting containers to talk to each other is half the job; the other half is the outside world. Outbound, containers reach the internet through the host, with pf handling NAT — translating their private addresses into something the internet will route back. Inbound, published ports get redirected from the host down into whichever container is supposed to answer them. Clean, symmetrical, sensible.

But this model carries one genuinely sharp edge, and I want you to remember it because it will save you an afternoon someday. A pf redirect does not fire for traffic that a node sends to its own public address. Read that twice. It means a service sitting on the same box has to be addressed by its container IP directly — going out to the public IP and expecting to loop back in simply won’t work. That one gotcha is the secret explanation behind an entire genre of baffling “but it works from outside, why does it fail from the box itself?” mysteries. And here’s the nice thing: once you know exactly where pf rules do and don’t apply, the network stops feeling like magic you’re at the mercy of, and starts feeling like a map you can actually read.