Here is the question every “highly available” system should have to answer out loud: what actually happens when a machine dies? Not in a diagram — in reality, with real traffic flowing, when you walk over and pull the plug. We did exactly that to this cluster, and the story of what broke, what held, and what healed itself is the most honest thing on this whole site.
Pulling the plug (the right way)
The first lesson came before the test even started. The tempting way to fake a node failure is to stop the containers on it — but that turns out to be a trap, because ocifbsd’s lifecycle can’t restart a stopped container in place, and cleaning up drags you into a fragile rebuild. The honest way is also the simpler one: power the whole virtual machine off. So that’s what we did — vm stop on one of the three nodes, a genuine “the server just died” event, reversible with a single vm start when we were done.
What held: traffic failover
With a third of the cluster suddenly gone, the site didn’t so much as flinch. Every request still came back 200, served by the two surviving nodes. The native ocifbsd load balancer noticed the dead backend, skipped it, and sent the connection to a replica that was alive. That’s failover, and it’s the baseline you’d hope for. But hoping isn’t knowing, and when we looked closely at the timing, we found something worth fixing.
The slow bleed, and the circuit-breaker
A dead machine doesn’t politely refuse your connection — it just goes silent. So every time the balancer tried the downed node, it had to wait out a two-second timeout before giving up and trying a live one. The request still succeeded, but it took two seconds instead of forty milliseconds. And because the balancer rotates through backends evenly, roughly one request in three kept paying that toll, over and over, for as long as the node stayed down. Correct, but bleeding.
The fix is an idea borrowed from electrical panels: a circuit-breaker. Give every backend a little shared health record. Each time a connection to it fails, count it; after three failures in a row, trip the breaker — mark that backend dead and stop even trying it for a ten-second cooldown. The measurement tells the whole story. Here’s the latency of twenty requests, in seconds, right after the node went down:
0.036 0.001 2.04 0.035 0.041 2.04 0.001 0.001 2.23 0.043 ...
└──────── three detections ────────┘
▲ breaker trips here
after: 0.043 0.001 0.042 0.040 0.001 0.039 0.036 0.043 ... (all fast)Three requests pay the two-second detection cost. Then the breaker trips, the dead node is short-circuited entirely, and every request afterward is fast again — while still returning correct content, because the traffic simply flows to the healthy replicas. The cost of a dead node dropped from “forever, on a third of requests” to “three requests, once.”
What didn’t heal: coming back from the dead
Then we powered the node back on, expecting it to slot back into the cluster. It didn’t. The virtual machine booted fine — but its containers stayed dark. Nothing on the node was responsible for starting them at boot, so the web server and app that had been running were simply… not. The machine was alive; the work it was supposed to be doing was not. That’s the difference between a server rebooting and a service recovering, and we’d been missing the second half.
The restart policy that was missing
So we built it: a small supervisor that gives every container a restart policy. It reads a manifest of what’s supposed to be running — each container’s image, its network address, and a policy of always, on-failure, or no — and a few times a minute it makes reality match. If a container that should be running isn’t, it recreates and restarts it, reapplying the network address it’s supposed to have. A crashed jail comes back within seconds. A rebooted node brings its whole workload back on its own, including re-establishing the pod network’s gateway so the node can talk to its containers again.
Crucially, it’s careful: a container that’s already healthy is left completely alone. We confirmed that by deploying the supervisor onto a node whose containers were happily running — same containers, same identities, nothing disturbed. It only acts when something is actually wrong.
The full loop
Put the pieces together and the cluster now survives a node death end to end, on its own. A machine dies: the circuit-breaker trips within three requests and short-circuits it, so visitors never feel it. The machine comes back: the supervisor restarts its containers and repairs its networking, and the load balancer’s breaker — which after its cooldown allows a single probe through — finds the node answering again and quietly folds it back into rotation. No pager, no human, no downtime the visitor could measure. We know because we watched it happen, plug and all.