/dev/vcio is not created when using a custom dtb on 64 bit RPiOS

[0n0w1c]

Describe the bug
When using a custom .dtb file, /dev/vcio is not created on boot.

To reproduce
Install RPiOS 64 bit beta
Update it, including the 5.10.63-v8+ kernel
Create a device tree from the running configuration (note: /dev/vcio does exist) with:
dtc -I fs -O dtb -o base.dtb /proc/device-tree
modify config.txt with device_tree=base.dtb
reboot
ls -l /dev/vcio

Expected behaviour
/dev/vcio is created

Actual behaviour
/dev/vcio is not created

System
Raspberry Pi 4 Model B Rev 1.2

Raspberry Pi reference 2021-05-07
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 071d3956353f2be964d2d337d3b123f1872b7f53, stage 2

Sep 28, 2021 11:27:35
Copyright (c) 2012 Broadcom
version 778b6a4f3c7d8d48bb63c02c47bcfbac79417bea (clean) (release) (start)

Linux raspberrypi 5.10.63-v8+ #1457 SMP PREEMPT Tue Sep 28 11:27:02 BST 2021 aarch64 GNU/Linux

Additional context
This same setup works as expected with the 5.10.63-v7 kernel and 32 bit RPiOS.

Note: The system info was typed not copied, so possible typos exist.

[0n0w1c]

I believe the issue is here:
/linux/drivers/char/broadcom/vcio.c

np = of_find_compatible_node(NULL, NULL,
				     "raspberrypi,bcm2835-firmware");
if (!of_device_is_available(np))
		return -ENODEV;

vcio.fw = rpi_firmware_get(np);
if (!vcio.fw)
		return -ENODEV;

[pelwell]

Please upload the .dtb somewhere - Google Drive, etc. - so that I can download and analyse it.

[0n0w1c]

If you copy the base32.dtb to the 64-bit RPiOS, it will create /dev/vcio.

base32.dtb
base64.dtb

[pelwell]

I can reproduce the issue here - indeed, the rpi_firmware_get call is failing for reasons as yet undetermined.

[pelwell]

This appears to be a startup order problem, where the behaviour depends on the precise ordering of the nodes in the DTB. Apart from the fact that the UART is disabled in the 64-bit .dtb file, the ordering is the only difference.

In a successful start we see the mailbox being probed before the firmware, followed by vcio:

[    0.062026] rpi_firmware_init: ret 1 0
[    0.062086] rpi_firmware_init: ret 2 0
[    0.106753] bcm2835_mbox_probe:
[    0.106935] bcm2835_mbox_probe: 0
[    0.107820] rpi_firmware_probe(ffffff80408f3c00):
[    0.120649] rpi_firmware_probe(ffffff80408f3c00): OK
[    1.301066] vcio: vcio_init(/soc/firmware)
[    1.301097] vcio: A
[    1.301113] vcio: B

A failure case has the firmware probing before the mailbox, causing it to return -EPROBE_DEFER that will cause it to be probed again later. Unfortunately the vcio driver is initialised (there is no probe function - this isn't a platform driver) before the second, successful firmware probe, causing it to fail before it gets to point B in the init function. Return -EPROBE_DEFER doesn't help here - init functions don't appear to be retried.

[    0.062723] rpi_firmware_init: ret 1 0
[    0.062782] rpi_firmware_init: ret 2 0
[    0.109677] rpi_firmware_probe(ffffff80408f6400):
[    0.109726] rpi_firmware_probe(ffffff80408f6400): -> -517
[    0.110178] bcm2835_mbox_probe:
[    0.110347] bcm2835_mbox_probe: 0
[    1.307862] vcio: vcio_init(/soc/firmware)
[    1.307941] vcio: A
[    1.414695] rpi_firmware_probe(ffffff80408f6400):
[    1.415635] rpi_firmware_probe(ffffff80408f6400): OK

dtc has a sort option - -s - but unfortunately it makes both generated dtbs - base32 and base64 (*) - cause the vcio driver to fail.

I think we're fortunate not to have been hit by this problem sooner. In the longer term I think the vcio driver needs to be made a platform driver with a DT node under the firmware node - like firmware-clocks driver. That should sort out the ordering. In the short term, you should use base32.dtb - although you may want to regenerate it with the UART enabled.

(*) I'm using the names you gave the files, indicating how you generated them, but there is nothing 32-bit-specific about base32 and nothing 64-bit-specific about base64.

[0n0w1c]

Thank you for the detail, it was educational for me.

[pelwell]

See #4624 as a potential fix for the problem. It would be great if you could build your own kernel and confirm that it resolves the issue for you.

[0n0w1c]

Sorry for the delay, I will begin shortly.

[0n0w1c]

I compiled and installed, rebooted, recreated the .dtb, and rebooted... success!

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.63-v8 #1 SMP PREEMPT Wed Oct 13 17:31:02 CDT 2021 aarch64 GNU/Linux

pi@raspberrypi:~ $ ls -l /dev/vcio
crw------- 1 root root 10, 59 Oct 13 18:59 /dev/vcio

pi@raspberrypi:~ $ dmesg | grep -b3 -a3 vcio
12160-[    1.304507] bcm2835-aux-uart fe215040.serial: there is not valid maps for state default
12251-[    1.306699] iproc-rng200 fe104000.rng: hwrng registered
12310-[    1.307083] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
12404:[    1.307604] vcio soc:firmware:vcio: A
12445-[    1.307674] B
12462-[    1.308207] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000
12548-[    1.308631] cacheinfo: Unable to detect cache hierarchy for CPU 0
--
14700-[    1.409373] raspberrypi-firmware soc:firmware: Firmware hash is 778b6a4f3c7d8d48bb63c02c47bcfbac79417bea
14808-[    1.409921] uart-pl011 fe201000.serial: there is not valid maps for state default
14893-[    1.417404] bcm2835-aux-uart fe215040.serial: there is not valid maps for state default
14984:[    1.418041] vcio soc:firmware:vcio: A
15025-[    1.418112] B
15042-[    1.464334] pci 0000:00:00.0: enabling device (0000 -> 0002)
15106-[    1.464498] xhci_hcd 0000:01:00.0: xHCI Host Controller