% Running qemu-system-aarch64 with -accel hvf on aarch64-darwin % Michael Stone % August 31, 2021 # tl;dr I now have an easy way to virtualize aarch64-linux NixOS VMs on Apple Silicon. It looks like: ```bash curl https://hydra.nixos.org/job/nixpkgs/trunk/ubootQemuAarch64.aarch64-linux/latest/download-by-type/file/binary-dist > u-boot.bin curl https://hydra.nixos.org/job/nixos/release-21.05-aarch64/nixos.sd_image.aarch64-linux/latest/download-by-type/file/sd-image > sd.img.zst nix run nixpkgs#zstd -- -d sd.img.zst nix build github:mstone/qemu-m1 ./result/bin/qemu-system-aarch64 -m 1024 -M virt,accel=hvf,highmem=off -smp 4 -cpu host -nographic -bios ./u-boot.bin -device virtio-rng-pci sd.img ``` It boots fast too. Some sample output (edited for length): ```default ... U-Boot 2021.04 (Apr 05 2021 - 15:03:29 +0000) ... Starting kernel ... ... [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x00000000] [ 0.000000] Linux version 5.10.60 (nixbld@localhost) (gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.35.1) #1-NixOS SMP Wed Aug 18 06:59:19 UTC 2021 ... [ 1.349180] 8021q: adding VLAN 0 to HW filter on device enp0s1 [ 1.508310] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s1: link becomes ready ... <<< Welcome to NixOS 21.05.2734.74d017edb67 (aarch64) - ttyAMA0 >>> ... ``` # Overview [qemu](https://qemu.org) is the pre-eminent free software full system emulator and virtual machine monitor (VMM). It is therefore a valuable tool in the toolkit of any aspiring low-level Linux engineer. Key to qemu's success is its support for emulating, mediating (virtualizing), and controlling access to quite diverse hardware including hardware for accelerating CPU virtualization such as is exposed by CPU instruction set architecture extensions like Intel VT-x, VT-d, AMD SVM, and [ARMv8-A virtualization](https://developer.arm.com/documentation/102142/0100). On Linux-based hosts, the key software interface for using these accelerated CPU ISA virtualization capabilities is called [KVM](https://linux-kvm.org); however, on macOS, the key interface is called "HVF", which is short for [Hypervisor.framework](https://developer.apple.com/documentation/hypervisor). Qemu has, for some time, supported KVM on Linux and HVF on macOS, specifically, on `x86_64-darwin`. However, with the advent of Apple's M1 hardware, qemu support for hvf on `aarch64-darwin` is now *the* barrier standing between developers based on macOS and fully controllable, fast, and responsive qemu-based Linux VM development, e.g., for server development and testing. Next, in good news, [Alexander Graf](https://twitter.com/_alexgraf) has developed patches for qemu that implement support for hvf on aarch64-darwin: [hvf: Implement Apple Silicon Support](https://patchew.org/QEMU/20210519202253.76782-1-agraf@csgraf.de/), which I believe have already been integrated into several friendly qemu wrappers such as ACVM and UTM. Unfortunately though, at the time of this writing, only ~1/2 of Alex's patches have been merged so far into the latest release of upstream qemu, presently v6.1.0. Consequently, since I would personally rather run qemu myself via a command lines that I directly control, I'd like to find a lower-level way to use Alex's patches. Here's what I've come up with. # Patches In order to apply the rest of Alex's patches, it's necessary to, e.g., filter the mbox of patches from patchew with `mutt` (or your favorite mbox editor) before applying them to a suitable qemu git checkout with `git am`. I have done this and have uploaded the result to . # Building qemu Qemu has its own (meson-based) build system but it also has non-trivial dependencies so, rather than trying to build qemu fully from scratch in the usual way that a regular qemu developer would, I'd prefer follow my usual route of building qemu using nixpkgs [`qemu` packaging](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/virtualization/qemu/default.nix) with whatever modifications are required to build qemu 6.1 + Graf's remaining patches. These changes are stored here in the [flake.nix](https://github.com/mstone/qemu-m1/blob/m1/flake.nix) and [flake.lock](https://github.com/mstone/qemu-m1/blob/m1/flake.lock) files that I've added until, hopefully, all these changes are applied upstream (both to qemu and to nixpkgs as is relevant)! # Codesigning One final complication remains that may be worth discussing which is "codesigning". On recent versions of macOS, a special endorsement called an "entitlement" is required in order to use `Hypervisor.framework`. Fortunately, qemu already contains patches to use Apple's `codesign` tool to add the required [entitlement](https://gitlab.com/qemu-project/qemu/-/blob/master/accel/hvf/entitlements.plist). Unfortunately, `nix`'s macOS stdenv does not provide `codesign`; hence, in order to fully automate the build using nix/nixpkgs, I had to piece together how to emulate `codesign`'s behavior, in this case using the unreleased [`entitlements` branch](https://github.com/thefloweringash/sigtool/tree/entitlements) of [Andrew Childs](https://github.com/thefloweringash)'s `sigtool`. # Running qemu-system-aarch64 Let's take apart the commands that I proposed in the [`#tl;dr`](./#tl;dr) section: ```bash curl https://hydra.nixos.org/job/nixpkgs/trunk/ubootQemuAarch64.aarch64-linux/latest/download-by-type/file/binary-dist > u-boot.bin curl https://hydra.nixos.org/job/nixos/release-21.05-aarch64/nixos.sd_image.aarch64-linux/latest/download-by-type/file/sd-image > sd.img.zst nix run nixpkgs#zstd -- -d sd.img.zst nix build github:mstone/qemu-m1 ./result/bin/qemu-system-aarch64 -m 1024 -M virt,accel=hvf,highmem=off -smp 4 -cpu host -nographic -bios ./u-boot.bin -device virtio-rng-pci sd.img ``` First, we have two commands for fetching recent NixOS aarch64-linux builds of [U-Boot](https://www.denx.de/wiki/U-Boot) and the [NixOS sd image](https://hydra.nixos.org/job/nixos/release-21.05-aarch64/nixos.sd_image.aarch64-linux/latest/). Next, we decompress the sd_image file, which is compressed with Facebook's [`zstd` compression suite](http://facebook.github.io/zstd/). Finally, we run our (signed!) `qemu-system-aarch64` VMM + device emulator binary. Some notes on the relevant flags: 1. `-m 1024` specifies how much RAM to give to our VM. 2. `-M virt,...` tells qemu a specific combination of hardware to emulate for use by our VM. 3. `...,accel=hvf,...` tells qemu to use `Hypervisor.framework` to make everything go fast. 4. `...,highmem=off` is required at least for now. See for details (or, alternately, for a patch that turns the feature off automatically when appropriate.) Note: If you mess this up, qemu will print this error message and halt: ```default qemu-system-aarch64: VCPU supports less PA bits (36) than requested by the memory map (40) ``` 5. `-smp 4` specifies how many cores to simulate. 6. `-cpu host`, as I currently understand things, tells qemu what instruction set to virtualize (or, in other situations, emulate?). This is a source of multiple hard-to-debug error messages. Some notes: * `-cpu max` starts the cpu, runs U-Boot, jumps to the kernel, and then hangs, requiring us to kill qemu externally or, more gracefully, to use the qemu monitor hotkeys to the same effect (`Ctrl-a x`). * Dropping the `-cpu` flag boots, but 10x more slowly. * `-cpu cortex-a57` also boots quickly, but produces a VM with fewer-than-expected CPU flags: ```default $ lscpu Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: ARM Model: 0 Model name: Cortex-A57 Stepping: r1p0 BogoMIPS: 48.00 NUMA node0 CPU(s): 0-3 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 fphp asimdhp cpuid dit ``` vs (with `-cpu host`): ```default $ lscpu Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: 0x00 Model: 0 Stepping: 0x0 BogoMIPS: 48.00 NUMA node0 CPU(s): 0-3 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atom ics fphp asimdhp cpuid asimdrdm jscvt fcma lrcp c dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb dcpodp flagm2 frint ``` 7. `-nographic` directs both qemu's monitor and emulated serial console / UART -- conventionally /dev/ttyAMA0 on ARM systems -- to stdio for convenience, but can be left off if you're comfortable working with qemu's graphical front-ends (including switching from the qemu monitor to the serial console output display) 8. `-bios u-boot.bin` tells qemu to use our U-Boot binary as a bootloader. 9. `-device virtio-rng-pci` tells qemu to provide an emulated hardware random number generator for use by the guest. 10. `sd.img` tells qemu to arrange for U-Boot to jump to our NixOS sd image. # Other notes ## codesign In case you'd like to do the codesigning using Apple's tool instead of `sigtool` or if you'd like to inspect the job `sigtool` did, here's a handy pair of commands that might help: ```bash # print existing entitlements codesign -d --entitlements :- ./result/bin/qemu-system-aarch64-unsigned # add hypervisor entitlement sudo codesign -s - --entitlements entitlements.xml --force ./result/bin/qemu-system-aarch64-unsigned ``` ## devicetree In case you want to inspect the device tree blob that qemu is generating to use to inform (in this case, U-Boot and Linux) what (virtual) hardware is available, you might try something like: ```bash qemu-system-aarch64 -machine virt,dumpdtb=virt.dtb -cpu host nix run nixpkgs#dtc -- -I dtb -O dts virt.dtb ```