I now have an easy way to virtualize aarch64-linux NixOS VMs on Apple Silicon. It looks like:
curl https://hydra.nixos.org/job/nixpkgs/trunk/ubootQemuAarch64.aarch64-linux/latest/download-by-type/file/binary-dist > u-boot.bin
curl https://hydra.nixos.org/job/nixos/release-21.05-aarch64/nixos.sd_image.aarch64-linux/latest/download-by-type/file/sd-image > sd.img.zst
nix run nixpkgs#zstd -- -d sd.img.zst
nix build github:mstone/qemu-m1
./result/bin/qemu-system-aarch64 -m 1024 -M virt,accel=hvf,highmem=off -smp 4 -cpu host -nographic -bios ./u-boot.bin -device virtio-rng-pci sd.imgIt boots fast too. Some sample output (edited for length):
...
U-Boot 2021.04 (Apr 05 2021 - 15:03:29 +0000)
...
Starting kernel ...
...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x00000000]
[ 0.000000] Linux version 5.10.60 (nixbld@localhost) (gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.35.1) #1-NixOS SMP Wed Aug 18 06:59:19 UTC 2021
...
[ 1.349180] 8021q: adding VLAN 0 to HW filter on device enp0s1
[ 1.508310] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s1: link becomes ready
...
<<< Welcome to NixOS 21.05.2734.74d017edb67 (aarch64) - ttyAMA0 >>>
...qemu is the pre-eminent free software full system emulator and virtual machine monitor (VMM). It is therefore a valuable tool in the toolkit of any aspiring low-level Linux engineer.
Key to qemu’s success is its support for emulating, mediating (virtualizing), and controlling access to quite diverse hardware including hardware for accelerating CPU virtualization such as is exposed by CPU instruction set architecture extensions like Intel VT-x, VT-d, AMD SVM, and ARMv8-A virtualization.
On Linux-based hosts, the key software interface for using these accelerated CPU ISA virtualization capabilities is called KVM; however, on macOS, the key interface is called “HVF”, which is short for Hypervisor.framework.
Qemu has, for some time, supported KVM on Linux and HVF on macOS,
specifically, on x86_64-darwin.
However, with the advent of Apple’s M1 hardware, qemu support for hvf
on aarch64-darwin is now the barrier standing
between developers based on macOS and fully controllable, fast, and
responsive qemu-based Linux VM development, e.g., for server development
and testing.
Next, in good news, Alexander Graf has developed patches for qemu that implement support for hvf on aarch64-darwin: hvf: Implement Apple Silicon Support, which I believe have already been integrated into several friendly qemu wrappers such as ACVM and UTM.
Unfortunately though, at the time of this writing, only ~1/2 of Alex’s patches have been merged so far into the latest release of upstream qemu, presently v6.1.0.
Consequently, since I would personally rather run qemu myself via a command lines that I directly control, I’d like to find a lower-level way to use Alex’s patches.
Here’s what I’ve come up with.
In order to apply the rest of Alex’s patches, it’s necessary to,
e.g., filter the mbox of patches from patchew with mutt (or
your favorite mbox editor) before applying them to a suitable qemu git
checkout with git am.
I have done this and have uploaded the result to https://github.com/mstone/qemu-m1.
Qemu has its own (meson-based) build system but it also has
non-trivial dependencies so, rather than trying to build qemu fully from
scratch in the usual way that a regular qemu developer would, I’d prefer
follow my usual route of building qemu using nixpkgs qemu
packaging with whatever modifications are required to build qemu 6.1
+ Graf’s remaining patches.
These changes are stored here in the flake.nix and flake.lock files that I’ve added until, hopefully, all these changes are applied upstream (both to qemu and to nixpkgs as is relevant)!
One final complication remains that may be worth discussing which is “codesigning”.
On recent versions of macOS, a special endorsement called an
“entitlement” is required in order to use
Hypervisor.framework.
Fortunately, qemu already contains patches to use Apple’s
codesign tool to add the required entitlement.
Unfortunately, nix’s macOS stdenv does not provide
codesign; hence, in order to fully automate the build using
nix/nixpkgs, I had to piece together how to emulate
codesign’s behavior, in this case using the unreleased entitlements
branch of Andrew
Childs’s sigtool.
Let’s take apart the commands that I proposed in the #tl;dr section:
curl https://hydra.nixos.org/job/nixpkgs/trunk/ubootQemuAarch64.aarch64-linux/latest/download-by-type/file/binary-dist > u-boot.bin
curl https://hydra.nixos.org/job/nixos/release-21.05-aarch64/nixos.sd_image.aarch64-linux/latest/download-by-type/file/sd-image > sd.img.zst
nix run nixpkgs#zstd -- -d sd.img.zst
nix build github:mstone/qemu-m1
./result/bin/qemu-system-aarch64 -m 1024 -M virt,accel=hvf,highmem=off -smp 4 -cpu host -nographic -bios ./u-boot.bin -device virtio-rng-pci sd.imgFirst, we have two commands for fetching recent NixOS aarch64-linux builds of U-Boot and the NixOS sd image.
Next, we decompress the sd_image file, which is compressed with
Facebook’s zstd
compression suite.
Finally, we run our (signed!) qemu-system-aarch64 VMM +
device emulator binary. Some notes on the relevant flags:
-m 1024 specifies how much RAM to give to our
VM.
-M virt,... tells qemu a specific combination of
hardware to emulate for use by our VM.
...,accel=hvf,... tells qemu to use
Hypervisor.framework to make everything go fast.
...,highmem=off is required at least for now. See https://patchwork.kernel.org/project/qemu-devel/patch/20201126215017.41156-9-agraf@csgraf.de/#23800615
for details (or, alternately, for a patch that turns the feature off
automatically when appropriate.) Note: If you mess this up, qemu will
print this error message and halt:
qemu-system-aarch64: VCPU supports less PA bits (36) than requested by the memory map (40)-smp 4 specifies how many cores to
simulate.
-cpu host, as I currently understand things, tells
qemu what instruction set to virtualize (or, in other situations,
emulate?). This is a source of multiple hard-to-debug error messages.
Some notes:
-cpu max starts the cpu, runs U-Boot, jumps to the
kernel, and then hangs, requiring us to kill qemu externally or, more
gracefully, to use the qemu monitor hotkeys to the same effect
(Ctrl-a x).
Dropping the -cpu flag boots, but 10x more
slowly.
-cpu cortex-a57 also boots quickly, but produces a
VM with fewer-than-expected CPU flags:
$ lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: ARM
Model: 0
Model name: Cortex-A57
Stepping: r1p0
BogoMIPS: 48.00
NUMA node0 CPU(s): 0-3
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 fphp
asimdhp cpuid ditvs (with -cpu host):
$ lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: 0x00
Model: 0
Stepping: 0x0
BogoMIPS: 48.00
NUMA node0 CPU(s): 0-3
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atom
ics fphp asimdhp cpuid asimdrdm jscvt fcma lrcp
c dcpop sha3 asimddp sha512 asimdfhm dit uscat
ilrcpc flagm sb dcpodp flagm2 frint-nographic directs both qemu’s monitor and emulated
serial console / UART – conventionally /dev/ttyAMA0 on ARM systems – to
stdio for convenience, but can be left off if you’re comfortable working
with qemu’s graphical front-ends (including switching from the qemu
monitor to the serial console output display)
-bios u-boot.bin tells qemu to use our U-Boot binary
as a bootloader.
-device virtio-rng-pci tells qemu to provide an
emulated hardware random number generator for use by the guest.
sd.img tells qemu to arrange for U-Boot to jump to
our NixOS sd image.
In case you’d like to do the codesigning using Apple’s tool instead
of sigtool or if you’d like to inspect the job
sigtool did, here’s a handy pair of commands that might
help:
# print existing entitlements
codesign -d --entitlements :- ./result/bin/qemu-system-aarch64-unsigned
# add hypervisor entitlement
sudo codesign -s - --entitlements entitlements.xml --force ./result/bin/qemu-system-aarch64-unsignedIn case you want to inspect the device tree blob that qemu is generating to use to inform (in this case, U-Boot and Linux) what (virtual) hardware is available, you might try something like:
qemu-system-aarch64 -machine virt,dumpdtb=virt.dtb -cpu host
nix run nixpkgs#dtc -- -I dtb -O dts virt.dtb