One container is a peaceful thing. It sits there, it does its job, and nobody has to think very hard about it. But the moment you have two containers that need to talk to each other — a web server that needs a database, the oldest pairing in the book — something sneaky happens. You are quietly, without signing up for it, no longer in the container business. You’re in the networking business. And getting containers on FreeBSD to reach each other, reach out to the wider internet, and be reachable from the outside world was the least glamorous, most consequential work behind this entire site.
VNET: giving every container its own little internet
The bedrock of all of this is VNET, FreeBSD’s network-stack virtualization. It’s a wonderful piece of engineering, and the idea is easy to fall in love with. Instead of every container elbowing for room on the host’s shared interfaces, each one gets its own stack: its own epair (think of it as a virtual patch cable with two ends), its own routing table, its own private view of the network world. A container isn’t a process squatting on the host’s address anymore. It’s a real network citizen with its own front door.
ocifbsd takes all those virtual interfaces and plugs them into a shared bridge — the pod network — so that containers living on the same node all sit on one friendly little subnet (here it’s 10.88.0.0/24) and can call each other by address, directly, no fuss. This is why VNET is non-negotiable for this project. It’s the difference between containers that merely coexist and containers that can genuinely cooperate.
The bugs that live in the seams
Now, if you’ve done any networking, you already know where the bugs hide. Never in the middle of things — always in the handoffs, the seams, the little moments where one component passes responsibility to another. This project was a textbook case, and I want to walk you through it because the bugs are genuinely instructive.
First, containers were coming up dutifully attached to the bridge — but without a working default route. So packets would cheerfully make it onto the subnet and then just… stop, like a traveler who reached the airport but never got a boarding pass. The fix was to actually apply the configured gateway when the interface comes up. Second, the network configuration was being written down correctly but not overlaid at container start, so a fresh run would boot up wearing yesterday’s settings; re-applying the netcfg overlay at launch sorted that out. And then even humble loopback got its say: services that expected to find themselves at 127.0.0.1 inside their own container needed the loopback interface configured too, not just the fancy external one. Every single one of these fixes was only a few lines of code. Every single one of them cost an afternoon to find. That’s networking for you.
From the little subnet out to the whole world
Getting containers to talk to each other is half the job; the other half is the outside world. Outbound, containers reach the internet through the host, with pf handling NAT — translating their private addresses into something the internet will route back. Inbound, published ports get redirected from the host down into whichever container is supposed to answer them. Clean, symmetrical, sensible.
But this model carries one genuinely sharp edge, and I want you to remember it because it will save you an afternoon someday. A pf redirect does not fire for traffic that a node sends to its own public address. Read that twice. It means a service sitting on the same box has to be addressed by its container IP directly — going out to the public IP and expecting to loop back in simply won’t work. That one gotcha is the secret explanation behind an entire genre of baffling “but it works from outside, why does it fail from the box itself?” mysteries. And here’s the nice thing: once you know exactly where pf rules do and don’t apply, the network stops feeling like magic you’re at the mercy of, and starts feeling like a map you can actually read.