Debian 11, 12, 13 not booting under propolis-server unless cpuid is provided

[taspelund]

I've been using falcon to spin up topologies that use helios and debian 11/12/13 images, and recently the debian images have been failing to boot.

This seems to be specifically happening on my machine, as @rcgoodfellow and CI do not seem to have the same problems as me.

When the guest fails to boot, the symptom is that the console displays the GRUB menu and auto-selects the default boot option after 3s, then an EFI Stub message is posted:

EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path

Example Debian 13 guest:
linux: Linux violin 6.12.38+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.38-1 (2025-07-16) x86_64 GNU/Linux

Information about my system:
CPU: AMD Ryzen 9 7900X
illumos: SunOS korgano 5.11 helios-2.0.23508 i86pc i386 i86pc

trey@korgano 10:46:16 PM | ~/debian-no-worky  main
‣ pfexec cpuid -d -c 1
CPU 1:
CPUID 00000000:00 = 00000010 68747541 444d4163 69746e65 | ....AuthcAMDenti
CPUID 00000001:00 = 00a60f12 02180800 7ef8320b 178bfbff | .........2.~....
CPUID 00000002:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 00000003:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 00000004:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 00000005:00 = 00000040 00000040 00000003 00000011 | @...@...........
CPUID 00000006:00 = 00000004 00000000 00000001 00000000 | ................
CPUID 00000007:00 = 00000001 f1bf97a9 00405fce 10000010 | ........._@.....
CPUID 00000007:01 = 00000020 00000000 00000000 00000000 |  ...............
CPUID 00000008:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 00000009:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 0000000a:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 0000000b:00 = 00000001 00000002 00000100 00000002 | ................
CPUID 0000000b:01 = 00000005 00000018 00000201 00000002 | ................
CPUID 0000000c:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 0000000d:00 = 000002e7 00000980 00000988 00000000 | ................
CPUID 0000000d:01 = 0000000f 00000980 00001800 00000000 | ................
CPUID 0000000d:02 = 00000100 00000240 00000000 00000000 | ....@...........
CPUID 0000000e:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 0000000f:00 = 00000000 000000ff 00000000 00000002 | ................
CPUID 0000000f:01 = 00000014 00000040 000000ff 00000007 | ....@...........
CPUID 00000010:00 = 00000000 00000002 00000000 00000000 | ................
CPUID 00000010:01 = 0000000f 00000000 00000004 0000000f | ................
CPUID 40000000:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000000:00 = 80000028 68747541 444d4163 69746e65 | (...AuthcAMDenti
CPUID 80000001:00 = 00a60f12 00000000 75c237ff 2fd3fbff | .........7.u.../
CPUID 80000002:00 = 20444d41 657a7952 2039206e 30303937 | AMD Ryzen 9 7900
CPUID 80000003:00 = 32312058 726f432d 72502065 7365636f | X 12-Core Proces
CPUID 80000004:00 = 20726f73 20202020 20202020 00202020 | sor            .
CPUID 80000005:00 = ff48ff40 ff48ff40 20080140 20080140 | @.H.@.H.@.. @..
CPUID 80000006:00 = 5c002200 6c004200 04006140 02009140 | .".\.B.l@a..@...
CPUID 80000007:00 = 00000000 0000003b 00000000 00006799 | ....;........g..
CPUID 80000008:00 = 00003030 791ef257 00005017 00010000 | 00..W..y.P......
CPUID 80000009:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 8000000a:00 = 00000001 00008000 00000000 1ebfbcff | ................
CPUID 8000000b:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 8000000c:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 8000000d:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 8000000e:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 8000000f:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000010:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000011:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000012:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000013:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000014:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000015:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000016:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000017:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000018:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000019:00 = f048f040 f0400000 00000000 00000000 | @.H...@.........
CPUID 8000001a:00 = 00000006 00000000 00000000 00000000 | ................
CPUID 8000001b:00 = 00000bff 00000000 00000000 00000000 | ................
CPUID 8000001c:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 8000001d:00 = 00004121 01c0003f 0000003f 00000000 | !A..?...?.......
CPUID 8000001d:01 = 00004122 01c0003f 0000003f 00000000 | "A..?...?.......
CPUID 8000001d:02 = 00004143 01c0003f 000007ff 00000002 | CA..?...........
CPUID 8000001d:03 = 0002c163 03c0003f 00007fff 00000001 | c...?...........
CPUID 8000001e:00 = 00000002 00000101 00000000 00000000 | ................
CPUID 8000001f:00 = 00000001 000000b3 00000000 00000000 | ................
CPUID 80000020:00 = 00000000 0000001e 00000000 00000000 | ................
CPUID 80000020:01 = 0000000b 00000000 00000000 0000000f | ................
CPUID 80000021:00 = 00062fcf 0000015c 00000000 00000000 | ./..\...........
CPUID 80000022:00 = 00000007 00084106 00000003 00000000 | .....A..........
CPUID 80000023:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000024:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000025:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000026:00 = 00000001 00000002 00000100 00000002 | ................
CPUID 80000027:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80000028:00 = 00000000 00000000 00000000 00000000 | ................
CPUID 80860000:00 = 00000000 00000000 00000000 00000000 | ................
CPUID c0000000:00 = 00000000 00000000 00000000 00000000 | ................

Moving falcon to point at this branch made it work:
oxidecomputer/falcon@070ab82

[iximeow]

strangely non-obvious to reproduce? my workstation is a 7950x and our relative CPUID bits are quite close:

# diff ~/trey <(cpuid -d -c 1)
3c3
< CPUID 00000001:00 = 00a60f12 02180800 7ef8320b 178bfbff | .........2.~....
---
> CPUID 00000001:00 = 00a60f12 02200800 7ef8320b 178bfbff | ...... ..2.~....
15c15
< CPUID 0000000b:01 = 00000005 00000018 00000201 00000002 | ................
---
> CPUID 0000000b:01 = 00000005 00000020 00000201 00000002 | .... ...........
28,29c28,29
< CPUID 80000002:00 = 20444d41 657a7952 2039206e 30303937 | AMD Ryzen 9 7900
< CPUID 80000003:00 = 32312058 726f432d 72502065 7365636f | X 12-Core Proces
---
> CPUID 80000002:00 = 20444d41 657a7952 2039206e 30353937 | AMD Ryzen 9 7950
> CPUID 80000003:00 = 36312058 726f432d 72502065 7365636f | X 16-Core Proces
34c34
< CPUID 80000008:00 = 00003030 791ef257 00005017 00010000 | 00..W..y.P......
---
> CPUID 80000008:00 = 00003030 791ef257 0000501f 00010000 | 00..W..y.P......
58c58
< CPUID 8000001d:03 = 0002c163 03c0003f 00007fff 00000001 | c...?...........
---
> CPUID 8000001d:03 = 0003c163 03c0003f 00007fff 00000001 | c...?...........

the differences on leaves 1, b, 80000002, 80000003, and 80000008 are all related to logical processor count (16-core * SMT = 32 == 20h vs 12-core * SMT = 24 == 18h) or brand string (7950x vs 7900x)

leaf 8000001d differs on NumSharingCache which is related to core count. but then leaf B doesn't get to the guest (I was noticing this in Omicron#8728) and leaf 8000001d is a synthetic topology so the host's values won't get to a guest anyway. the brand string makes it to the guest, and the core count in leaf 1 ebx is reset to the core count of the guest so the host's reading there shouldn't matter.

---

and so both expected and unexpected: things seem to work fine on my 7950x? I did a few things:

• cloned falcon and adjusted an example to use debian-13.1, this runs and cargo run exec violin whoami prints root so it definitely is up. This works on the 7950x with both 070ab82 (the explicit CPUID leaves) and e91c724
  before it which is what Trey saw fail to boot Debian guests
• used the propolis-server that Falcon downloads directly with propolis-cli and a standalone toml to run it
• used a propolis-server that I built from the same and an earlier commit I'd been hacking on

in all cases the Debian 13.1 image boots, and in the directly-using-propolis-server cases a Debian 11 image I usually validate Propolis against also boots.

One difference not reflected here would be guest-visible MSRs, but setting explicit CPUID bits shouldn't change that. But it's a point of possible difference between Trey's and my workstations.

I'd be really curious what we might see with a stuck guest via mdb -b <vmm uuid>, not sure how we'd get to a backtrace but at the very least getting some addresses and comparing to the guest's kernel might be interesting. based on seeing EFI stub: ... and not the linux banner at timestamp 00000.0 the kernel hasn't gotten to pr_notice("%s", linux_banner); so that's a relatively short window to be at..

edit: one other open question to me is, which stlouis commit is helios-2.0.23508 reflective of? my workstation is on debug bits that's probably off the branch directly but should be recent. just one place there might be divergence.

[iximeow]

mystery is solved! here's a dtrace script I'm gonna put in-tree in some form, imminently:

# cat cpuid-queries.d 
fbt::vcpu_emulate_cpuid:entry {
        this->rax = (uint64_t*)arg2;
        this->rbx = (uint64_t*)arg3;
        this->rcx = (uint64_t*)arg4;
        this->rdx = (uint64_t*)arg5;
        printf("CPUID query: rax = %08x, rcx = %08x", *this->rax, *this->rcx);
}

fbt::vcpu_emulate_cpuid:return {
        printf("returned leaves: rax = %08x, rbx = %08x, rcx = %08x, rdx = %08x",
                *this->rax,
                *this->rbx,
                *this->rcx,
                *this->rdx
        );
}

which shows the stuck VM suspiciously enumerating subleaves of CPUID leaf 8000_0026:

 21  69627         vcpu_emulate_cpuid:entry CPUID query: rax = 80000026, rcx = 004a0089
 21  69628        vcpu_emulate_cpuid:return returned leaves: rax = 00000001, rbx = 00000002, rcx = 00000100, rdx = 00000013
 21  69627         vcpu_emulate_cpuid:entry CPUID query: rax = 80000026, rcx = 004a008a
 21  69628        vcpu_emulate_cpuid:return returned leaves: rax = 00000001, rbx = 00000002, rcx = 00000100, rdx = 00000013
 21  69627         vcpu_emulate_cpuid:entry CPUID query: rax = 80000026, rcx = 004a008b
 21  69628        vcpu_emulate_cpuid:return returned leaves: rax = 00000001, rbx = 00000002, rcx = 00000100, rdx = 00000013
 21  69627         vcpu_emulate_cpuid:entry CPUID query: rax = 80000026, rcx = 004a008c
 21  69628        vcpu_emulate_cpuid:return returned leaves: rax = 00000001, rbx = 00000002, rcx = 00000100, rdx = 00000013
 21  69627         vcpu_emulate_cpuid:entry CPUID query: rax = 80000026, rcx = 004a008d
 21  69628        vcpu_emulate_cpuid:return returned leaves: rax = 00000001, rbx = 00000002, rcx = 00000100, rdx = 00000013
 21  69627         vcpu_emulate_cpuid:entry CPUID query: rax = 80000026, rcx = 004a008e
 21  69628        vcpu_emulate_cpuid:return returned leaves: rax = 00000001, rbx = 00000002, rcx = 00000100, rdx = 00000013
 21  69627         vcpu_emulate_cpuid:entry CPUID query: rax = 80000026, rcx = 004a008f
 21  69628        vcpu_emulate_cpuid:return returned leaves: rax = 00000001, rbx = 00000002, rcx = 00000100, rdx = 00000013
...

this was fixed by #915, but the cpuid-test branch both bumped Propolis which included that fix and included the specific Milan CPUID definition. Both mechanisms effectively hide leaf 8000_0026.

additional info: Trey's system using Falcon pointed to main did successfully boot Debian 11.0 images, is also where we reproduced the hang-at-boot behavior on Debian 13.1. using oxidecomputer/falcon@63fa97d alone, which bumps Propolis without the explicit Milan CPUID bits, also resulted in Debian 13.1 booting on Trey's machine. knowing this, I went back to 6b5734f just before #915 and exactly as you'd expect it hangs at the same spot on my 7950x as well.

we didn't try Debian 12, given that there's a pretty clear explanation of Debian 13 getting stuck at boot in this way.

less interesting Linux stuff, Debian 12 -> Debian 13 picks up Linux kernel 6.10 which has https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/cpu&id=63edbaa48a579bb167aa92aa5826888730819fa2. I'm pretty sure that the loop we see is parse_topology_leaf(..., 0x80000026) stuck on being told that (via rcx) that all subleaves are valid descriptions of topology level 0.