tsirysndr/bsdkrun

A Firecracker-style microVM launcher for BSD and Linux guests on macOS and Linux, built on libkrun (Hypervisor.framework/KVM) - tsirysndr/bsdkrun

1 stars · 0 forks · 0 watchers

README

bsdkrun

A Firecracker-style microVM launcher for BSD and Linux guests on macOS and Linux, built on libkrun (which drives Apple's Hypervisor.framework on macOS and KVM on Linux). bsdkrun is a thin, purpose-built CLI: it wraps libkrun's C ABI in a handful of safe Rust bindings and boots a guest three ways — from a UEFI firmware image (the guest's own EFI loader boots a normal disk), from a direct kernel + FDT (no bootloader), or straight from an OCI image (bsdkrun linux alpine pulls it from any registry, extracts the rootfs, and boots it like docker run). It is deliberately small: one FFI module, one CLI, no daemon.

Platforms: macOS on Apple Silicon (Hypervisor.framework) and Linux on amd64 or arm64 (KVM). A hardware-virtualized guest runs the host's CPU arch, so bsdkrun detects the arch and pulls the matching kernel, OCI image, and agent automatically. macOS is arm64-only; Linux works on both x86_64 and aarch64. FreeBSD boots via EFI on macOS and via PVH direct kernel on Linux/amd64; NetBSD direct-boots its kernel everywhere. The amd64 PVH boots need our PVH-enabled libkrun fork. (Linux support is new — see the KVM e2e CI.)

Contents

Why this exists Install Prerequisites Build Usage

probe freebsd / netbsd firmware kernel linux

Managing machines Networking Disks Console Preparing a guest image Project layout Troubleshooting Status License

Why this exists The usual microVM stacks (Firecracker, Cloud Hypervisor) don't run on macOS, and the usual macOS VM tooling (QEMU, vftool, UTM) isn't microVM-shaped. libkrun gives you a Firecracker-like "configure a context, then start_enter" model on top of Hypervisor.framework — its batteries are aimed at Linux guests, and bsdkrun both leans into that (running OCI images directly) and points the same machinery at FreeBSD / NetBSD guests.

FreeBSD: on macOS, boot via firmware/EFI — we hand libkrun its bundled EDK2 firmware and let the guest's loader.efi take over from the EFI System Partition on the disk (that firmware ships only with the macOS libkrun-efi). On Linux/amd64, boot via PVH direct kernel: bsdkrun downloads its bundled agent-injected UFS rootfs + FreeBSD's FIRECRACKER kernel (no ACPI, MPTable, virtio-mmio built in) and enters it at its PHYS32_ENTRY — which needs our PVH-enabled libkrun fork (see the FreeBSD notes). NetBSD: boot via direct kernel — no bootloader or firmware, so bsdkrun netbsd works on both macOS and Linux. On arm64 it uses NetBSD's evbarm GENERIC64 kernel + bsdkrun's agent-injected gzimg. NetBSD ships no amd64 disk image, so on amd64 bsdkrun uses its own bundled FFS rootfs plus the MICROVM kernel, entered via PVH — which needs our PVH-enabled libkrun fork (see the NetBSD notes). Linux: run any OCI image as a microVM — bsdkrun fetches a prebuilt kernel, pulls the image from any registry, extracts its rootfs, and boots it docker run-style, with internet access out of the box. See linux.

DragonFly BSD is out of scope: there's no arm64 port.

The open research question is guest-side virtio-mmio device discovery. libkrun exposes its virtio devices over MMIO (there is no PCI bus), and describes them via the ACPI/FDT it hands the guest. Whether a given BSD kernel enumerates those virtio-mmio devices — and routes its console to libkrun's virtio-console — is exactly what this tool is for probing.

Install macOS (Apple Silicon) — a prebuilt, already-signed binary via Homebrew: brew install tsirysndr/tap/bsdkrun This auto-taps libkrun/krun and pulls in its dependencies (libkrun, gvproxy). The binary ships codesigned with the hypervisor entitlement, so there's nothing else to set up — jump to Usage. npm — install the prebuilt host binary for your platform (macOS/arm64, Linux/x64, Linux/arm64): npm install -g @bsdkrun/cli # or: npx @bsdkrun/cli linux alpine -- echo hi A postinstall step downloads the matching bsdkrun from the GitHub release and verifies its SHA-256. On Linux the archive bundles libkrun (libkrun.so/libkrunfw.so, rpath'd to $ORIGIN), so it works with no separate libkrun install — only gvproxy is needed for guest networking. On macOS it's just the binary and links Homebrew's libkrun (brew install libkrun). Unsupported platforms (Windows, Intel macOS, 32-bit) fail the install with a clear message. See npm/ for details. Nix flake — builds bsdkrun with all its dependencies. On Linux (amd64/arm64) it links nixpkgs' libkrun; on macOS it links your Homebrew libkrun, so those need --impure (brew install libkrun/krun/libkrun first) and produce a binary re-signed with the hypervisor entitlement.

Linux — needs /dev/kvm access: sudo usermod -aG kvm $USER && newgrp kvm

macOS (Apple Silicon) — impure link against Homebrew's libkrun

Prerequisites You need libkrun, a Rust toolchain (rustup default stable; edition 2021), and access to the hypervisor. The hypervisor part differs by OS. macOS (Apple Silicon) libkrun, krunvm, and krunkit live in the libkrun/krun tap (redirected from the old slp/krun). Homebrew 6.x requires you to trust a third-party tap before it will run its install code: brew tap libkrun/krun brew trust libkrun/krun # required on Homebrew 6.x for third-party taps brew install libkrun krunkit

libkrun provides libkrun.dylib (the C ABI we link against). krunkit ships the EDK2 UEFI firmware we use for EFI boot (.../share/krunkit/KRUN_EFI.silent.fd).

The Hypervisor entitlement (the part that bites everyone). A binary that calls libkrun must be codesigned with com.apple.security.hypervisor (plus com.apple.security.cs.disable-library-validation so it can load the Homebrew dylibs). Without it, krun_create_ctx/krun_set_vm_config succeed but krun_start_enter fails at VM creation with Internal(Vm(VmSetup(VmCreate))) / errno 22 (EINVAL). Worse, every cargo build strips the codesignature, so you must re-sign after each build — the Makefile does this for you (entitlements in bsdkrun.entitlements). Linux (amd64 or arm64, KVM) libkrun uses KVM on Linux — no codesigning, but you need /dev/kvm access. Add yourself to the kvm group (or run under sudo): sudo usermod -aG kvm $USER && newgrp kvm Ubuntu has no libkrun package, so build it (and its bundled kernel, libkrunfw) from source — see the KVM e2e workflow for the exact steps: git clone --depth 1 https://github.com/containers/libkrunfw && make -C libkrunfw && sudo make -C libkrunfw install git clone --depth 1 https://github.com/containers/libkrun && make -C libkrun && sudo make -C libkrun install sudo ldconfig build.rs finds libkrun via pkg-config libkrun (or the standard lib dirs; override with LIBKRUN_PREFIX=/path). There's nothing to sign — make build skips the codesign step on Linux. Some BSD image-prep steps (losetup/mount) need root, and bsdkrun runs them with sudo automatically when needed.

On Linux the CI boots the linux (OCI), netbsd, and freebsd (both direct-kernel, PVH) paths — on x86_64 only, since GitHub's arm64 runners have no /dev/kvm; arm64-on-Linux (which reuses the aarch64 kernel + agent) is validated on a KVM-capable host. The BSD-under-KVM amd64 boots go via our PVH libkrun fork, which the CI builds.

Build make build # cargo build (debug) [+ codesign on macOS] make release # cargo build --release [+ codesign on macOS] make run ARGS="..." builds and runs in one step.

⚠️ macOS: don't run cargo build then the binary directly — it'll be unsigned and fail at boot with errno 22. Go through make, or re-run make sign after a bare cargo build. On Linux there's nothing to sign, so cargo build is fine (the make sign steps are no-ops there).

The build.rs locates libkrun via brew --prefix libkrun (macOS) or pkg-config (Linux), override with LIBKRUN_PREFIX=/path, and embeds an rpath so the shared library resolves at runtime.

Usage bsdkrun [--log-level N] <command>

Global: --log-level N log verbosity, 0=off .. 5=trace (default 1)

--log-level drives two things at once: libkrun's internal logging and bsdkrun's own logging (via the tracing crate). Both go to stderr, so they never mix with the guest console on stdout. 0→warn, 1..3→info, 4→debug, 5→trace. Set the RUST_LOG environment variable to override bsdkrun's filter with anything EnvFilter accepts (e.g. RUST_LOG=bsdkrun=debug). For a clean guest console, use --log-level 0 or 1. probe — sanity check the toolchain Verifies that libkrun links and that a context can be created and configured. Does not boot (so it won't exercise the hypervisor entitlement — see the note above). make run ARGS="probe" freebsd / netbsd — one-liner BSD microVMs The quickest way to a BSD guest — fetches the image (and, for NetBSD, the kernel) if needed, then boots it: bsdkrun freebsd # bundled FreeBSD image (agent baked in): EFI on macOS, PVH on Linux/amd64 bsdkrun freebsd --version 14.3 # official FreeBSD 14.3 VM image from download.freebsd.org bsdkrun netbsd -d # NetBSD-current in the background; prints its id bsdkrun netbsd --version 10.1 -d --port 2222:22 Like bsdkrun linux, you can pass a command to run inside the guest after --. The guest boots, its agent runs the command (streaming stdout/stderr), and — without -d — the VM powers off afterward, with bsdkrun exiting on the command's status (a one-shot, à la docker run). With -d the machine is left running once the command returns. Needs networking (the agent), so it's incompatible with --no-net: bsdkrun freebsd -- uname -a # boot, run, print, power off; exit = command's status bsdkrun netbsd -- sh -c 'sysctl hw.model' bsdkrun freebsd -d -- pkg install -y curl # run a setup step, then leave the VM up With no command (and no -d), a foreground bsdkrun freebsd / bsdkrun netbsd drops you straight into an interactive shell over the agent — the bundled images are headless (no console login), so this is the way in — and powers the VM off when you exit, like foreground bsdkrun linux. A short ⋯ waiting for the guest agent line shows while the guest boots (~15-20s). bsdkrun freebsd # boot, then an interactive /bin/sh; exit (Ctrl-D) powers it off bsdkrun netbsd -d # no shell — just background it and print the id (use exec/ssh/shell) Add --verbose to stream the guest's boot console to stdout while it comes up (instead of the terse spinner) — handy for watching a boot or asserting on it in CI. The command output / shell follows once the agent is up: bsdkrun freebsd --verbose -- uname -a # full boot log, then the command output, on stdout Both carry the usual machine options (-d, --persist, -v/--volume, --version, --attach-disk, --port, --cpus/--mem), so per-machine CoW disk clones and ps/logs/shell/stop all apply. They differ in how they boot:

freebsd boots differently per host OS:

On macOS it wraps fetch + firmware: it auto-locates libkrun's KRUN_EFI firmware (via $BSDKRUN_FIRMWARE, a local images/KRUN_EFI.fd, or krunkit's Homebrew install; --firmware overrides). By default (on arm64) it downloads bsdkrun's bundled image with the guest agent pre-installed, so exec/shell work out of the box; pass --version X to boot an official FreeBSD VM image from download.freebsd.org instead (no agent — you'd install it manually). On Linux/amd64 it PVH-direct-boots bsdkrun's bundled agent-injected UFS rootfs + the FreeBSD FIRECRACKER kernel (no ACPI, MPTable enumeration, virtio-mmio + serial built in; built from source by the image workflow). Needs the PVH libkrun fork; override the kernel command line with $BSDKRUN_FREEBSD_CMDLINE.

netbsd wraps fetch + a direct kernel boot — no firmware — so it works on macOS and Linux.

firmware — boot a disk through its UEFI loader The explicit form (the freebsd/netbsd shortcuts wrap it). Point it at libkrun's EDK2 firmware and a raw disk image that carries an EFI System Partition: make release ./target/release/bsdkrun firmware \ --firmware "$(brew --prefix)/share/krunkit/KRUN_EFI.silent.fd" \ --disk images/fbsd15.raw \ --cpus 2 --mem 2048

Use libkrun's own KRUN_EFI.silent.fd firmware — not a generic QEMU AAVMF/edk2 build. libkrun uses its own guest memory layout, and only its firmware matches it. The .silent variant just suppresses EDK2's own console chatter; the guest OS console still comes through the serial console (see Console below). The disk is attached read-write as a virtio-blk device (block_id = "root").

This path is confirmed booting FreeBSD 15 / arm64 all the way to a login: prompt. Two things make it work: bsdkrun wires the guest's serial console to your terminal (see below), and the FreeBSD image needs a one-line console hint on its ESP (see Preparing a FreeBSD arm64 image). kernel — boot a kernel directly (no bootloader) The path for NetBSD/evbarm and bare kernel experiments. libkrun generates the FDT and jumps into the kernel: ./target/release/bsdkrun kernel \ --kernel path/to/netbsd \ --format elf \ --cmdline "console=..." \ --disk images/root.raw \ --cpus 1 --mem 512 --format is one of elf (default) or raw. --initramfs is optional. linux — run an OCI image as a microVM Run any Docker Hub / OCI image as a Linux microVM, docker run-style. bsdkrun fetches a prebuilt aarch64 kernel (cached), pulls the image for linux/arm64, extracts its rootfs, and boots it:

run alpine's default shell

run a specific command, with more RAM

any registry / tag

Kernel — a prebuilt aarch64 vmlinux is downloaded from vmlinux-builder and cached. libkrun's aarch64 loader needs the raw Image format, so bsdkrun flattens the ELF to an Image (in pure Rust — no binutils) and caches that too. Pick a release with --kernel-version (default 7.1.5), or point at your own kernel with --kernel /path (ELF or raw Image). Rootfs — the OCI image is pulled straight from the registry (no Docker daemon; just curl + tar) and its layers are extracted, applying whiteouts. The result is cached, content-addressed by image digest, so a repeat run is instant. By default the rootfs is packed into an initramfs and booted from RAM, with a generated /init that mounts /proc + /sys, configures networking, and runs the image's Entrypoint/Cmd (honoring Env/WorkingDir). Entrypoint — Docker semantics: args after -- replace the image Cmd; --entrypoint replaces the Entrypoint. When the workload exits, the VM powers off cleanly. Networking — on by default (see Networking); the guest gets internet access (ICMP/DNS/TCP) via gvproxy, configured through the kernel command line so the image itself needs no ip/dhcp tools. --no-net disables it.

Notes:

Rootfs — by default the rootfs is served from disk over virtio-fs (no RAM-size limit, so it's fine for large images). bsdkrun clones the cached rootfs per machine with an APFS copy-on-write clone (cp -Rc — instant, no extra disk until the guest writes), so machines stay isolated and the shared image cache stays pristine. It boots our own init from the shared root (not libkrun's init.krun, which only works with the bundled libkrunfw kernel). Needs a guest kernel with CONFIG_VIRTIO_FS=y (note: CONFIG_FUSE_FS alone is not enough — the default prebuilt kernel has it). --initramfs boots from an initramfs instead (the whole rootfs is loaded into RAM). Use it for a kernel without virtio-fs. Then size --mem above the image size — bsdkrun warns if it looks too small. Either way the image must have a /bin/sh (scratch/distroless images won't boot this way), and the console defaults to hvc0 (libkrun's virtio-console; --console overrides it).

systemd — turn an OCI guest into a full systemd system OCI images boot with bsdkrun's tiny generated /init as PID 1 — great for docker run-style workloads, but no services, journal, or timers. One command flips a debian/ubuntu/fedora guest to real systemd: id=$(bsdkrun linux -d -v dev debian -- sleep infinity) bsdkrun systemd $id setup # installs systemd if missing + the agent unit, marks the rootfs bsdkrun stop $id id=$(bsdkrun linux -d -v dev debian) # same volume -> boots systemd as PID 1 bsdkrun systemd $id status # "PID 1: systemd" setup installs systemd where missing (apt-get/dnf — Alpine has no systemd and fails with a clear message), writes + enables a bsdkrun-agent.service unit (so exec/shell keep working under systemd), and drops a marker (/etc/bsdkrun-systemd) that makes the generated init exec systemd as PID 1 on the next boot. disable removes the marker. In systemd mode the image entrypoint/-- command is not run — systemd owns userspace; manage workloads as units. Boot on a volume (-v) so the installed packages + marker survive across machines.

Managing machines bsdkrun keeps a small SQLite database (sqlx) under $XDG_STATE_HOME/bsdkrun recording the machines you run, the images you've pulled, and disks you've attached — each with a Docker-style short id. The same Docker-like commands work for every guest type — Linux (linux) and BSD (firmware / kernel) alike — bsdkrun records each machine's kind and applies the right logic:

run any machine in the background; prints its id

bsdkrun ps # list running machines (-a for all, incl. exited) bsdkrun images # list images: pulled OCI images + fetched BSD images bsdkrun logs $id # print the machine's console log bsdkrun logs -f $id # follow it live bsdkrun exec $id uname -a # run a command inside the guest (-t for a PTY, -e K=V for env) bsdkrun shell $id # open an interactive shell in the guest bsdkrun stop $id # stop a running machine Any unique id prefix works (bsdkrun stop 8e1c). shell attaches to the guest console: for a Linux machine that's an interactive shell (with exit/re-attach); for BSD it's the guest's own console (e.g. the login: prompt). Copy-on-write disks — a BSD machine's root disk is cloned per machine with an APFS clonefile (cp -c — instant, and costs no extra disk until the guest writes), so you can boot many microVMs from one base image concurrently without touching it. Pass --persist to boot the disk in place instead (writes persist; one machine at a time). Linux machines get the same isolation via their per-machine virtio-fs clone. Persistent volumes (-v NAME) — by default every boot starts from a fresh clone, so guest changes are thrown away when the machine exits. To keep them across reboots, name a volume — works the same for Linux, FreeBSD and NetBSD: bsdkrun linux -d -v web alpine # persistent Linux rootfs bsdkrun freebsd -d -v db # persistent FreeBSD disk Reuse the same -v NAME and the machine comes back up with your changes intact. It's a single writer at a time (run one machine per volume), and it's mutually exclusive with --persist (which writes to the base image itself). The mechanism is the same idea for every guest — a copy-on-write clone that persists:

Linux serves the volume as a writable virtio-fs root. First use copy-on-write clones the OCI image into the volume dir (clonefile on APFS / reflink on btrfs/xfs — instant, and only grows as the guest writes; a plain copy elsewhere); later boots reuse it, so your changes persist. Needs the default virti

source