Every confident claim on this site — that the cluster is real, that your session follows you around, that a load balancer written from scratch in FreeBSD can hold a genuine crowd — is worth exactly nothing until something angry tries to knock it flat. Talk is free. Load is not. So we stopped talking and started swinging. What follows is the log of the day we pointed a steadily growing swarm of concurrent visitors at the three-node cluster and watched, patiently, for the first crack to appear.
The rig on the table
No mockups, no simulators, no comfortable lies. The target was the clustered deployment exactly as you’re reading it right now: three FreeBSD nodes — fb16-1, fb16-2, fb16-3 — each running an identical image of Nginx (with a FastCGI cache out front) sitting ahead of PHP-FPM and WordPress, all wrapped in native OCI jails managed by ocifbsd. Standing guard in front of the whole thing is the ocifbsd proxy: that protocol-agnostic layer-4 balancer baked right into the runtime, spreading connections round-robin and quietly failing dead backends over to the next. Behind the nodes, one shared Redis holds every visitor’s session, and one shared MariaDB is the single source of truth. The load hit the same TLS front door, the same proxy, and the same jails that serve real traffic on any ordinary day. Nothing was staged.
Phase one: 125 at once, and the cache earns its paycheck
We opened gently — if you can call 125 simultaneous clients hammering the front page and the inner pages “gentle.” And crucially, every one of those clients wasn’t just checking that a response came back. It was checking that the bytes it got were the correct bytes: not a truncated page, not an error dressed up in a 200’s clothing, not garbage. The verdict: 100% correct content. No wrong pages, no corruption, not one byte out of place. That’s the boring result you desperately want.
But the number that made everyone lean in was latency — and this is where the FastCGI cache stepped up and did something dramatic. A cold WordPress render is a lot of work: PHP boots, MariaDB gets interrogated, the page is assembled brick by brick. Under contention, that cold render clocked in around 5.3 seconds. Painful. But with the cache keyed per-URL, configured to skip only on cookies, POSTs, and wp-admin, a warm hit came screaming back in roughly 63 milliseconds. That is not a typo, and I checked it twice myself. Caching the right things chopped the served latency by around eighty-fold. It’s an old lesson, but it’s worth relearning every single time it happens: the fastest database query in the world is the one you never had to make.
The session that flatly refused to get lost
A balancer that flings you to a new machine on every click is worse than useless if your session evaporates the moment it does. So right in the middle of the test, we kept an eye on one single session’s counter while the proxy deliberately knocked it around the cluster like a pinball. It climbed: 1 → 2 → 3 → 4 → 5 → 6. And every one of those increments was served by a different node, under the same session id. Because the session lives in shared Redis rather than on any one node’s local disk, it just quietly followed the visitor from fb16-1 to fb16-2 to fb16-3 and back around again, never missing. That’s the thing that makes the web tier genuinely stateless — and stateless is exactly the property that makes it scale sideways.
Phase two: crank it until something bends
Then we stopped being polite. We turned the dial up and up — past 250, past 500, past 1,000 — and finally parked an outright firehose of 2,000 concurrent connections on the cluster to go find the edge of the map. And — good news for the honesty of this post — we found it.
The genuinely interesting part wasn’t that it bent. Everything bends eventually. It was where. The proxy, notably, did not fall over — after one real fix (I’ll get to it) it just kept calmly accepting and distributing connections across every core, unbothered. The ceiling turned out to be the application tier: a finite pool of PHP-FPM workers, and behind them, that single shared MariaDB. The uncacheable requests — logins, POSTs, admin actions — can’t be short-circuited by the cache; they have to reach all the way to PHP and the database. And past a certain point, there simply aren’t enough workers or database connections to go around. At the 2,000 extreme, roughly nine in ten uncacheable requests were turned away or slowed rather than served instantly. That’s not the proxy failing. That’s the proxy being scrupulously honest about a backend that’s run out of hands.
And it’s worth being precise about which knob does what, because this is where a lot of intuition goes wrong. Throwing more RAM and CPU at the nodes raises the capacity — more workers, more cache headroom, a bigger number before things saturate. Genuinely helpful! But it does not change the shape of the limit. A single database of record and a fixed worker pool will always, always have a saturation point somewhere; you’re just moving it. The honest way to push the ceiling further up is more replicas, and eventually a database tier that scales out too. Knowing which knob addresses which limit — that’s the entire reason you run a stress test instead of guessing.
Two bugs the crowd shook loose
Here’s a thing they don’t tell you: a big enough crowd is a debugging tool. Two bugs that had been hiding in plain sight came tumbling out under the pressure.
- The half-closed connection. Under heavy load the proxy started handing back the occasional truncated response — maddening, because it was intermittent. The culprit was the pump loop tearing down both directions of a connection the moment either side signalled EOF. So when a client half-closed its upload while the response was still streaming back, the response got its sentence cut off mid-word. The fix was proper half-close handling: shut down only the finished direction with
SHUT_WR, and keep draining the other until it, too, is genuinely done. After that, the 5xx blips under load simply vanished. - The node that couldn’t reach itself. One replica’s WordPress kept throwing 500s, and the reason was delightfully sneaky: it was configured to reach the database at its own node’s public IP — and the packet-filter redirect that maps that public IP down to the database container does not apply to traffic a node originates to itself. The fix was to point that replica straight at the database’s container address on the pod network, sidestepping the redirect entirely. A small, very FreeBSD-flavored lesson about exactly where
pfrules do and do not fire.
What the whole exercise actually proved
Three things, concretely, and I’ll stand behind each one. First: the cluster serves correct content under real concurrency — not “approximately correct,” not “correct-ish,” but every byte. Second: sessions migrate, which means the web tier is truly stateless and any node can answer any request. Third — and this is the one I’m proudest of — the failure mode is understood and, frankly, boring. The app tier saturates predictably, the proxy stays honest about it, and the path to more headroom (more replicas, a scale-out data tier) is clearly signposted rather than shrouded in mystery. A system you can push to its absolute limit on purpose, and then calmly explain afterward, is a system you can actually trust in production. That’s the whole reason we turned the dial all the way to 2,000.