RC RANDOM CHAOS

Firecracker contains the attack it never sees

Firecracker microVMs on EC2 bare metal boot in under a second and tear down before any EDR, log flush, or forensic capture can observe them.

· 8 min read
Firecracker contains the attack it never sees

Firecracker runs as a userspace virtual machine monitor on top of KVM. KVM needs hardware virtualization extensions - VT-x on Intel, SVM on AMD. Standard EC2 instances run as guests on the Nitro hypervisor and do not expose those extensions to the guest. The only EC2 surface that does is bare metal. So “Firecracker inside EC2” means Firecracker on a .metal instance - c5.metal, m5.metal, i3.metal - where the host OS touches the silicon directly and /dev/kvm is real.

Each microVM is one Firecracker process. The jailer wraps it: chroot, cgroups, PID and network namespaces, a dropped UID, and a seccomp-BPF syscall filter applied before the guest vCPU runs. Boot to userspace is roughly 125 milliseconds. Memory overhead per microVM sits under 5 MiB. Layer a minimal Linux guest and a headless Chromium on top of that and a browser session is serving in under a second. AWS Lambda and Fargate run on the same primitive. The sub-second number is the product. It is also the security problem, and not for the reason the marketing implies.

Two boundaries decide whether this design holds. The first is guest-to-host isolation. The second is whether anything an attacker does inside a microVM is ever observed. The first is well-engineered. The second is broken by construction.

The guest-to-host boundary lives in device emulation. A VMM presents virtual hardware to the guest, and the guest’s drivers talk to that hardware across a shared-memory interface the VMM parses. That parser is the escape surface. The canonical case is VENOM, CVE-2015-3456, CVSS 7.7 - a buffer overflow in the QEMU virtual floppy disk controller. The guest wrote past the FDC’s fixed-size data buffer and reached host memory in the QEMU process. Guest-to-host code execution through a device nobody used and few remembered was attached. Firecracker’s entire design is a reaction to that lesson. There is no floppy controller, no BIOS, no PCI bus, no USB, no SCSI, no legacy emulation. The device model is virtio-net, virtio-block, virtio-vsock, a serial console, and a minimal i8042 controller that exists only to signal guest reset. Less surface, fewer parsers, fewer VENOMs.

What remains is virtio. Firecracker exposes virtio devices over the MMIO transport. The guest driver places descriptors into a virtqueue in shared guest memory, advances the available ring index, and the VMM reads those descriptors, walks the chain, performs the I/O, and writes the used ring. A hostile guest controls every field the VMM consumes - descriptor addresses, lengths, flags, ring indices, chain depth. The VMM has to validate that each descriptor address and length resolves inside the guest’s physical memory region, that index arithmetic wraps correctly, that a descriptor chain does not loop forever, that a length does not overflow when added to an offset. A missed bound or an integer truncation in that path lets the guest steer the VMM into reading or writing memory it should not. That is the corruption primitive on this platform.

Firecracker is written in Rust, which removes the spatial memory-safety class that produced VENOM. Safe Rust will not write past a slice. That guarantee does not cover everything. Index arithmetic on ring positions, conversion between guest physical and host virtual addresses, and the unsafe blocks that map guest memory are where logic and integer bugs survive the borrow checker. A confused read or write inside the Firecracker process is the start of a host-side primitive, the same shape of escalation a renderer-process corruption is in a browser chain. From there an attacker pivots toward code execution in the context of the Firecracker process. This maps to MITRE T1611, Escape to Host.

The seccomp-BPF filter is the second containment layer. Even with code execution inside the VMM, the process is restricted to a syscall whitelist tuned to what Firecracker legitimately needs. The jailer’s chroot, namespaces, and cgroups are the third. Below all of it sits the layer no language choice fixes: KVM itself. Every VMEXIT the guest triggers is handled by the host kernel KVM module in ring 0. A bug there is reachable from the guest regardless of how clean the userspace VMM is, and it is the deepest and most valuable escape target on the host. A microVM is exactly as isolated as the host kernel’s KVM implementation on the day it runs.

Cross-tenant exposure does not require an escape at all. Microarchitectural side channels cross the VM boundary through shared silicon. L1TF - Foreshadow, CVE-2018-3646 - lets a guest read data resident in the L1 data cache, including data belonging to another VM scheduled on the sibling hyperthread. The MDS family, CVE-2018-12130 among them, samples in-flight data from CPU internal buffers across the same boundary. On a .metal host packing thousands of microVMs from different trust domains onto shared cores, two sessions on sibling threads of one physical core can leak across that line. The mitigation is not a Firecracker patch. It is SMT disabled or core scheduling enforced, plus current microcode, applied by whoever operates the host. Skip it and the isolation story has a hole no VMM closes.

The real-world use of sub-second microVMs is not theoretical, and most of it is mundane. Ephemeral headless-browser fleets power scraping, automated QA, link unfurling, and screenshot services. The same primitive powers abuse infrastructure: ad fraud, credential stuffing, account-creation farms, and CAPTCHA-solving at volume. The economics are the point. A browser session that costs a fraction of a cent and lives under a second lets an operator run millions of attempts and discard every one. Cloudflare’s bot reporting tracks exactly this class of disposable browser automation hitting login and signup endpoints. Okta’s customer base absorbs the credential-stuffing side of it, sessions cycling through fresh identities faster than any per-host reputation system can score them. In ATT&CK terms the pattern runs T1583 to acquire the infrastructure, T1078.004 when valid cloud credentials spin the fleet on someone else’s account, T1496 when the capacity is hijacked compute, and T1564 - Hide Artifacts - because the ephemerality is defense evasion as a property of the architecture, not a step the attacker takes.

That ephemerality is where detection collapses. Consider what an EDR or SIEM assumes: that a host lives long enough to enroll an agent, complete a TLS handshake to a collector, establish a behavioral baseline, and flush a buffer. A microVM with a sub-second lifetime satisfies none of those preconditions. Agent enrollment does not finish. The first log flush, on a multi-second interval, never fires before teardown. The guest image is minimal by design - no auditd shipping, no Sysmon equivalent wired to a pipeline unless the platform operator baked one in and engineered it to emit before destruction. Inside the microVM, the default telemetry is nothing.

The host view is no better. From the .metal instance, every microVM is one more firecracker process. At scale that is thousands of identical short-lived process trees - jailer forks firecracker, opens /dev/kvm, issues KVM ioctls, creates and tears down a TAP device - repeating continuously. To a host EDR this is high-cardinality noise that looks the same whether the session rendered a marketing page or brute-forced a login. Separating a malicious session from a legitimate one requires per-session correlation the platform has to build itself, because the OS does not draw that line. Network telemetry inherits the same gap. Guest egress NATs out through the host ENI, so VPC flow logs and any external observer see the host’s IP set, not the session behind it. Attribution to an individual microVM exists only if the platform tagged the flow on the way out.

Then the VM is destroyed and the evidence goes with it. The rootfs is typically a read-only base image plus a discarded overlay or ephemeral block device. Guest RAM is anonymous host memory, freed on teardown. No disk image, no memory capture, no shell history, no process record survives unless something grabbed it mid-flight. This is anti-forensics achieved by architecture rather than by an attacker wiping logs. There is nothing to wipe. An incident responder arrives to a host that ran the activity and retains none of it.

One telemetry seam survives the teardown, and it is not on the host. It is the control plane. The fleet does not schedule itself - an orchestration layer authenticates, requests capacity, and drives the lifecycle, and those API calls are logged where the microVMs are not. On AWS that is CloudTrail: the RunInstances and scaling calls, the IAM principal behind them, the credential that spun the metal, the region selected. Detection migrates off the endpoint and onto that record. The signal is not a process inside a guest that lived 800 milliseconds. It is the rate, identity, and geography of the calls that manufacture guests, and the anomaly in a credential creating capacity it never created before. T1078.004 leaves a mark in the control plane even when the workload leaves none on the host.

The patch boundary on the VMM is straightforward - Firecracker ships fixes as tagged releases, and the affected version range is whatever predates the fix for a given virtio or vsock advisory. Patching the VMM is the easy part. The residual exposure after that patch is three things the release notes do not close. The host KVM surface, owned by the kernel version, not Firecracker. The microarchitectural side channels, owned by host SMT configuration and microcode. And the observability gap, owned by the platform operator and closed by no patch at all, because the speed that sells the service is the same speed that outruns every tool built on the assumption that a machine lives long enough to be watched. The sub-second boot is not the vulnerability. It is the condition under which the instrumentation never gets a turn.

Share

Keep Reading

Stay in the loop

New writing delivered when it's ready. No schedule, no spam.