I have been excited about containers since I first started using Docker, around 2014 or 2015. The talk that got me there was Solomon Hykes at Twitter University, where the idea that stuck was containers as a unit of software delivery: package the application and everything it depends on into one thing, then run that thing anywhere. I still send that talk to anyone new to containers, because it explains how the project came to be rather than just how to use it.

Lately I have been paying closer attention to containers again, because agent sandboxing is all the rage and containers looked like the natural way to get a sandbox. Then Ivan Velichko’s article on the OCI runtime spec showed up on my X feed, and it pulled a thread I did not expect.

I already knew AWS Lambda runs on Firecracker microVMs, and I had accepted the usual line that VMs isolate better than containers. I believed it without being able to explain why. This is me actually working that out.

What I had wrong

My mental model was simple: processes on Linux are isolated from each other, namespaces and cgroups make containers more isolated still, and I roughly equated that isolation with being secure.

The isolation part is true. Namespaces, cgroups, and seccomp are kernel features, and the kernel really does enforce them. When a process in a container asks what else is running on the machine, the kernel answers with a filtered list. None of that is fake.

What I had collapsed together is that isolated and secure against hostile code are not the same claim. The isolation is bookkeeping that one kernel instance keeps about many processes: what each one may see, how much each may consume. Break the kernel and it stops meaning anything, because the thing enforcing the boundary is the thing you just compromised.

A quick refresher on the terms

A handful of words come up constantly below, so here they are up front.

Hypervisor and VMM (virtual machine monitor) mean roughly the same thing in everyday use: software that creates and runs virtual machines. A hypervisor takes a physical host with CPU, RAM, and network, storage attached alongside, and slices it into virtual pieces: vCPUs, vRAM, vNETs. You hand those out as separate machines, and each one believes it owns real hardware. Some clouds call them VSIs, virtual server instances. AWS says EC2 instance. The detail that matters later is that every one of those machines boots its own kernel.

A hypervisor slices a physical host’s CPU, RAM, and network into vCPU, vRAM, and vNET, handing them to VMs that each boot their own guest kernel

The one place those two words come apart is KVM. KVM is the hypervisor, a module inside the Linux kernel that uses the CPU’s virtualization extensions (Intel VT-x, AMD-V) to run guest code directly on the real CPU instead of emulating it. Firecracker is a VMM, a userspace process that asks KVM to create a VM, gives it memory and a small set of virtual devices, and boots a guest kernel inside it. Firecracker does not do the virtualization itself. It drives KVM, which does.

On the container side there are three kernel features, each answering a different question:

  • Namespaces: what can this process see? Its own process list, its own network interfaces, its own filesystem tree. Visibility only. No limits on consumption.
  • cgroups: how much can it use? Caps on CPU, memory, and I/O. Limits only. No effect on what it can see.
  • seccomp: which syscalls may it make at all? Linux has 350 or so. Most programs need a few dozen. Blocking the rest shrinks the attack surface.

Docker, containerd, and Kubernetes are built almost entirely on those three. Which is why calling a Linux container a lightweight VM is not quite right: there is no VM in the picture at all, just a process the kernel treats specially. It is shorthand, much like calling a container just a Linux process.

One kernel, many tenants

On a Kubernetes node, every container is a process on the same kernel instance. Every pod, every workload, every customer if you are multi tenant. Namespaces give each one its own view, cgroups cap what each consumes, and neither creates a second kernel or a second syscall table.

That matters because of how much surface a kernel exposes: every one of those syscalls, the filesystem code, the network stack, the namespace and cgroup machinery itself, dozens of driver subsystems. All of it reachable in principle from inside a container. If any of it has an exploitable bug, and the history of container escape CVEs says that is not hypothetical, then hostile code inside a container may be able to escalate privileges or break out to the host. When that happens every other container on the box is exposed too, because nothing stronger than the shared kernel’s bookkeeping was protecting them.

One kernel per tenant

Firecracker starts somewhere else. Each tenant gets their own guest kernel, booted inside a KVM backed virtual machine, and Firecracker runs as an ordinary host process, one per microVM.

A microVM is a real virtual machine, just a stripped down one, and that is what makes it quick to boot. So both containers and microVMs end up described as lightweight, for opposite reasons: a container skips virtualization altogether, while a microVM keeps it and trims everything around it.

QEMU is the other well known VMM, and the contrast is the point. QEMU carries a large general purpose device model: USB, VGA, BIOS emulation, a long list of emulated hardware. Firecracker carries almost none of that, just enough for a guest kernel plus network, block storage, and a serial console. That is a startup time win and, as below, a security win.

KVM is a module inside the host kernel acting as the hypervisor, while Firecracker is a userspace process acting as the VMM that asks KVM to create the VM. QEMU’s guest facing device model is much larger than Firecracker’s

The isolation here is not bookkeeping, it is hardware. KVM uses the CPU’s virtualization extensions to set up a second level of page tables (Intel calls it EPT, AMD calls it NPT) mapping guest memory into a private region of host RAM. The CPU’s memory management unit enforces that mapping on every memory access. It is not a promise the kernel keeps in software. A guest cannot address memory outside its own mapping.

Which leads to the question that made this click for me. If hostile code fully compromises its own guest kernel and gets root inside the microVM, what has it won? Its own disposable single tenant kernel. Nobody else is using it, and it gets thrown away with the microVM.

Containers share one host kernel behind a thin namespace boundary, so one kernel bug exposes every container. MicroVMs each get their own guest kernel behind a hardware enforced EPT boundary, so compromising one wins only a disposable single tenant kernel

One bug versus two

This is the part worth being careful with, because the overselling is easy.

MicroVMs are not more secure because some layer is unbreakable. KVM has had guest escape vulnerabilities. Firecracker’s device model is still code and code has bugs. The real claim is about how many independent bugs an attacker needs and how different they have to be.

On a container host, one bug is enough. A single vulnerability in the shared kernel, in namespace handling or a driver path or cgroup accounting, is reachable from inside any container, and compromising it compromises everything sharing that kernel.

On a Firecracker host you need two, and they have to be different in kind:

  1. A bug in the guest kernel or the app in it, to get code execution inside the microVM.
  2. A separate bug in KVM or in Firecracker’s device model, to escape the guest and reach the host or another guest.

Those are not two rolls of the same dice. They are two different attack surfaces that both have to fail, and the second one is deliberately small. Firecracker’s guest facing surface is a handful of virtual devices instead of a sprawling device model, and KVM is a narrow, heavily audited part of the kernel that every cloud provider has strong incentive to harden, because so much multi tenant infrastructure sits on it.

So the defensible version of “microVMs are more secure” is defense in depth where the second layer is smaller and more scrutinized than the first. Two locks on different doors from different manufacturers beats one lock, even though neither lock is unbreakable.

On a container host one bug in the shared kernel reaches the host and every other container. On a Firecracker host an attacker needs two independent bugs, one in the guest kernel and a second in KVM or the device model

The container primitives come back on top

In production you can wrap each Firecracker process in its own cgroup, chroot, and seccomp filter on the host. Firecracker ships a component called jailer for exactly this.

That is worth pausing on, because it is the same set of primitives containers are built from, and here they are being used to contain the VMM itself. The VM boundary does the heavy lifting, and then container style sandboxing sits around it as a second, outer layer. If something goes wrong inside Firecracker’s own userspace code, short of a full escape through KVM, it is still stuck inside a cgroup, a chroot, and a syscall filter before it can reach anything else on the host.

So containers are not the wrong tool. They are one of the layers.

Two caveats

The hardware is still shared. MicroVMs on one physical host share CPU cores, caches, and memory controllers below any software boundary. Side channel attacks of the Spectre and Meltdown family are not solved by containers or by microVMs. That is a separate problem both models have to handle on their own.

Containers are not insecure. If everything on the host belongs to the same trust boundary, your own services and your own team’s workloads, namespaces and cgroups are enough. You are preventing accidental interference, not defending against an adversary. It only starts to matter when the code you are running came from somewhere you do not trust.

Where I landed

I reached for containers in the first place because they isolate, and somewhere along the way I had quietly decided that isolation was the same as security. It is not. Namespaces and cgroups give me a process that cannot see or starve its neighbors, which is real and genuinely useful, but every one of those neighbors is still standing on the same kernel. That boundary is something the kernel maintains, and it holds for exactly as long as the kernel does.

MicroVMs move the boundary somewhere harder to argue with. Each tenant gets its own kernel and the CPU enforces the separation. Neither model is unbreakable, and that was never the claim. The difference is that getting out of a container takes one bug in a large shared surface, while getting out of a microVM takes two unrelated bugs, one of them in a surface that has been kept deliberately small.

So the question was never which one is secure. It is how much I need to trust the code I am about to run. Inside a single trust boundary, containers answer that fine and always did. When the code is untrusted, the distance between one bug and two is the whole point.

Resources