Blog

One container is a peaceful thing. It sits there, it does its job, and nobody has to think very hard about it. But the moment you have two containers that need to talk to each other — a web server that needs a database, the oldest pairing in the book — something sneaky happens. You are quietly, without signing up for it, no longer in the container business. You’re in the networking business. And getting containers on FreeBSD to reach each other, reach out to the wider internet, and be reachable from the outside world was the least glamorous, most consequential work behind this entire site.

VNET: giving every container its own little internet

The bedrock of all of this is VNET, FreeBSD’s network-stack virtualization. It’s a wonderful piece of engineering, and the idea is easy to fall in love with. Instead of every container elbowing for room on the host’s shared interfaces, each one gets its own stack: its own epair (think of it as a virtual patch cable with two ends), its own routing table, its own private view of the network world. A container isn’t a process squatting on the host’s address anymore. It’s a real network citizen with its own front door.

ocifbsd takes all those virtual interfaces and plugs them into a shared bridge — the pod network — so that containers living on the same node all sit on one friendly little subnet (here it’s 10.88.0.0/24) and can call each other by address, directly, no fuss. This is why VNET is non-negotiable for this project. It’s the difference between containers that merely coexist and containers that can genuinely cooperate.

The bugs that live in the seams

Now, if you’ve done any networking, you already know where the bugs hide. Never in the middle of things — always in the handoffs, the seams, the little moments where one component passes responsibility to another. This project was a textbook case, and I want to walk you through it because the bugs are genuinely instructive.

First, containers were coming up dutifully attached to the bridge — but without a working default route. So packets would cheerfully make it onto the subnet and then just… stop, like a traveler who reached the airport but never got a boarding pass. The fix was to actually apply the configured gateway when the interface comes up. Second, the network configuration was being written down correctly but not overlaid at container start, so a fresh run would boot up wearing yesterday’s settings; re-applying the netcfg overlay at launch sorted that out. And then even humble loopback got its say: services that expected to find themselves at 127.0.0.1 inside their own container needed the loopback interface configured too, not just the fancy external one. Every single one of these fixes was only a few lines of code. Every single one of them cost an afternoon to find. That’s networking for you.

From the little subnet out to the whole world

Getting containers to talk to each other is half the job; the other half is the outside world. Outbound, containers reach the internet through the host, with pf handling NAT — translating their private addresses into something the internet will route back. Inbound, published ports get redirected from the host down into whichever container is supposed to answer them. Clean, symmetrical, sensible.

But this model carries one genuinely sharp edge, and I want you to remember it because it will save you an afternoon someday. A pf redirect does not fire for traffic that a node sends to its own public address. Read that twice. It means a service sitting on the same box has to be addressed by its container IP directly — going out to the public IP and expecting to loop back in simply won’t work. That one gotcha is the secret explanation behind an entire genre of baffling “but it works from outside, why does it fail from the box itself?” mysteries. And here’s the nice thing: once you know exactly where pf rules do and don’t apply, the network stops feeling like magic you’re at the mercy of, and starts feeling like a map you can actually read.

Here’s a thing that’s easy to take for granted: where do container images actually come from? For a lot of people the answer is a shrug and a pull command — images arrive from somewhere far away as finished, sealed black boxes, and you just trust that whatever’s inside is fine. The images behind this site don’t work like that. They’re not delivered; they’re assembled, right here on FreeBSD, from a recipe short enough to read over a cup of coffee. This is the story of ocifbsd build — the command that lets the platform construct the very containers it later turns around and runs.

A recipe you’ve probably read before

If you’ve ever written a Dockerfile, the recipe here — we call it a Containerfile — is going to feel like running into an old friend. There’s a FROM line to pick your starting point. There are RUN steps to install and configure software. There’s COPY to bring your own files aboard, and ENV, WORKDIR, and CMD to describe how the finished thing ought to wake up and start working. Same grammar, same muscle memory.

The difference is what’s underneath the words. The base here is the official FreeBSD OCI image, so when a RUN pkg install line goes fetching nginx, php, mariadb, or redis, it’s pulling honest-to-goodness FreeBSD packages — the real thing, compiled for this system, not Linux binaries wearing a clever disguise and hoping nobody checks their ID. What you install is what you get.

What really happens when you hit “build”

Let’s peek behind the curtain, because this is where it gets satisfying. Each RUN step doesn’t just execute in some vague sandbox — it runs inside a chroot of the image as it exists so far, with devfs mounted so the tools have the device nodes they quietly expect to find. When the step finishes, that devfs gets unmounted again, cleanly, no mess left behind. Step by step, the image grows.

And building on top of an image that already exists — the classic “I just want to rebuild this thing” scenario — is handled with the same care. Files carrying the immutable schg flag are gently unwound first, and any stale devfs mounts left over from before are torn down before the new build begins. The upshot is that a second build is every bit as clean as the first one; you don’t accumulate weird ghosts from previous attempts. What pops out the other end is a completely normal ocifbsd image — tagged, stored under the local registry path, and ready to run, indistinguishable from anything you might have pulled from elsewhere.

Why building it here, on-platform, actually matters

You could reasonably ask: who cares where the image is built, as long as it runs? Here’s the case for caring. A container platform that can only consume images somebody else made is, honestly, only half a platform. It’s a player that can’t cook, only microwave. By building on FreeBSD, from FreeBSD packages, the entire supply chain stays native and stays inspectable. There’s no cross-build step to squint at, no emulation layer humming along in the background, and no mystery meat about what’s actually inside the thing you’re about to expose to the internet.

So when the Machine Room dashboard casually reports that the WordPress jail is running local/wordpress:latest, that’s not a label pointing off into the distance. That image was born on this same machine, from a recipe living in the repository, assembled by the very same tool that’s now supervising it as it serves you this page. The site, quite literally, builds itself — and then, because we thought you’d enjoy it, turns around and tells you all about it.

Let me tell you a small secret about the page you’re looking at: it’s here to win an argument. Not a loud one — there’s no manifesto, no flame war — just a quiet, stubborn claim that a lot of people assume is false. The claim is this: you don’t need Linux, and you don’t need a big background daemon shepherding everything, to run real, modern, containerized applications. You need an operating system that already knows how to fence off processes, hand out private networks, and count who’s using what. FreeBSD has known how to do all three for years. Everything you’re reading right now is served from exactly that stack, and it’s the evidence.

So what’s actually humming away underneath?

On the surface, nothing exotic. This is an ordinary WordPress site: PHP-FPM does the thinking, MariaDB remembers everything, Redis keeps the fast-moving bits close at hand, and Nginx greets you at the door. If you’ve run a website before, you know this cast of characters. What’s unusual is the stage they’re standing on. Every one of those services lives inside a native OCI container built and babysat by ocifbsd, a runtime that takes the Open Container Initiative model — the same shape of images and containers you’d recognize anywhere — and maps it cleanly onto FreeBSD’s own building blocks.

Here’s the fun part, the bit that makes an old FreeBSD hand grin. An image becomes a jail. A pod network becomes a VNET — a genuine, private network stack — wired up to a bridge. A resource limit stops being a polite suggestion and becomes a RACCT rule the kernel enforces directly. There’s no compatibility shim in the middle, no layer quietly pretending to be Linux so the tools upstairs don’t panic. There’s nothing to pretend to be. It’s FreeBSD all the way down, doing what it was already good at.

Two shapes, one story

The same application runs here in two different shapes, and I’d encourage you to think of them as two chapters of one story rather than two separate demos. The first shape is a single FreeBSD VM where all the containers share one host — compact, tidy, the entire platform folded into a single box you could carry in your pocket if pockets held virtual machines. The second is a three-node cluster with a native load balancer standing out front, a shared session store so a visitor can bounce from node to node without noticing, and one database of record sitting calmly behind them all.

The single site is the “does this even work?” answer. The cluster is the “okay, but does it scale?” answer. Both are live, both are public over HTTPS, and — this is my favorite detail — both can describe themselves. Open the dashboard under “The Machine Room” and you’re watching real resource accounting streamed out of the very jails that are, at that exact moment, rendering the page you’re reading. The site isn’t telling you a story about some other system. It’s pointing a camera at itself.

Why go to all this trouble?

Fair question. The honest answer is that the container ecosystem grew up assuming one particular kernel, and that quiet assumption cost the BSDs a seat at a table they were, frankly, architecturally ready for. Jails predate a good chunk of the isolation machinery the container world would later reinvent with great fanfare. So the thing that was missing was never capability. It was ergonomics — the smooth pull, build, run, network, observe rhythm that people now simply expect from a container platform. That gap, that missing sense of “oh, this just feels nice to use,” is precisely what ocifbsd sets out to close.

The rest of these posts are the guided tour. We’ll teach containers to talk to each other over the network, drop a load balancer into the toolbox, make sessions follow you from node to node, and then — because talk is cheap — deliberately try to knock the whole thing over with a stress test. You can start anywhere; each post stands on its own. Just know that whatever page you land on, it’s turtles all the way down. Jails, really. But you get the idea.

Welcome to WordPress. This is your first post. Edit or delete it, then start writing!