Stress test, in detail: where the cluster actually breaks

We already told the story of pushing this cluster until it bent. This is the follow-up with the receipts: a fresh, methodical stress test where we turned the concurrency dial up in stages, wrote down every number, and — most importantly — figured out which part of the stack gives way first. The short version: the load balancer we built has enormous headroom, and the two real limits are exactly the ones you’d predict once you see them.

How the test was run

For the duration of the test we flipped Cloudflare’s proxy off, so the load hit our own origin directly instead of being absorbed by a CDN — then flipped it back on when we were done. Then we drove traffic with hey at rising concurrency levels, against two very different kinds of request: the cached front page (served straight from Nginx’s FastCGI cache) and an uncacheable page that has to wake up PHP-FPM and query MariaDB on every hit. Those two numbers tell completely different stories, which is the whole point.

Finding 1 — the app tier: ~30 renders a second, then 502s

Uncacheable requests are the honest test of the application, because every single one travels the full path: Nginx → PHP-FPM → MariaDB → back again. Here’s what happened as we raised concurrency (each level held for 12 seconds, fresh connection per request):

concurrency   throughput   slowest request   result
    50          31 rps         3.2 s          all 200 OK
   100          31 rps         4.8 s          all 200 OK
   200          29 rps         8.8 s          all 200 OK
   400          27 rps        17.4 s          all 200 OK  (deep queue)
   800         229 rps        26.0 s          933 x 200  +  7093 x 502  <-- break

The application tier holds a steady ~30 uncacheable renders per second across the three nodes. Notice throughput barely moves from 50 to 400 concurrent — it's saturated the whole time — while the latency climbs and climbs as requests pile into the queue: three seconds, then five, then nine, then seventeen. At 800 concurrent the queue finally overflows and the upstream starts refusing work: 88% of requests come back as 502 Bad Gateway. That's the break. It isn't mysterious — it's a finite pool of PHP-FPM workers and one shared database, doing exactly as much as they can and no more.

Finding 2 — the cache: 30× the throughput, until the edge gives

The cached front page is a different universe. With reused connections, the public site served this:

concurrency   throughput   average latency   result
   200         861 rps        0.17 s          all 200 OK
  1000          54 rps       16.5 s           all 200, but crawling
  3000         100 rps       18.1 s           all 200, but crawling

861 requests a second at moderate concurrency — roughly thirty times the dynamic ceiling, which is the entire reason the FastCGI cache exists. But look what happens past a thousand concurrent connections: throughput falls off a cliff and latency balloons to sixteen seconds, even though serving a cached page costs almost nothing. When near-free work grinds to a halt, the bottleneck isn't the work — it's the connections themselves. Something in front of the cluster was choking on the sheer number of simultaneous sockets. So we went looking for it.

Finding 3 — the load balancer is not the bottleneck

To find the real ceiling we cut out the public path entirely and pointed the load straight at the ocifbsd proxy on the internal network — no TLS, no home-internet gateway in the way. Same cached request, same tool, same machine generating load:

concurrency   throughput   average latency   result
    50        1308 rps        0.04 s          all 200 OK
   200         892 rps        0.21 s          all 200 OK
  1000        1012 rps        0.54 s          all 200 OK   <-- public path did 54 here
  3000         462 rps        1.6 s           all 200 OK
  6000         234 rps        9.8 s           all 200 OK, zero errors

There's the answer. Hit directly, the proxy does over a thousand requests a second at a thousand concurrent connections — right where the public path had collapsed to fifty-four — and it keeps returning correct responses with zero errors all the way up to six thousand concurrent. The native ocifbsd load balancer has headroom to spare. The high-concurrency collapse we saw from the outside lives above it: the TLS-terminating ingress and the consumer-grade gateway's NAT table, which is simply not built to juggle thousands of simultaneous connections. In production that layer is exactly what a CDN like Cloudflare is for — which is why it sits in front.

What actually breaks, and what to do about it

• Dynamic content saturates at ~30 req/s and hard-fails near 800 concurrent. This is the shared PHP-FPM + single MariaDB. The fix is horizontal: more web replicas raise the worker count, and eventually the database tier has to scale out too. More RAM and CPU raise the number; they don't change the shape.
• Cached content does ~860 req/s through the front door and would do far more if the front door were bigger. The cache is doing its job — the ceiling here is the edge, not the app.
• The ocifbsd proxy sustains 1000+ req/s at 1000 concurrent and never errors up to 6000. The piece we built ourselves is the piece with the most room. That's a good place to be.
• Throughout all of it, when a request failed it failed cleanly — a 502, not corrupted content — and the circuit-breaker and failover kept the healthy paths fast. A system that degrades honestly under overload is one you can actually reason about.

None of these limits is a surprise, and that's the best thing we can say about them. We can point at the exact request rate where each tier tops out, explain why, and name the knob that moves it. That's what a stress test is for — not to prove nothing breaks, but to know precisely what breaks, when, and what to do next.