/dev/vcio is not created when using a custom dtb on 64 bit RPiOS #4620

0n0w1c · 2021-10-07T18:05:18Z

Describe the bug
When using a custom .dtb file, /dev/vcio is not created on boot.

To reproduce
Install RPiOS 64 bit beta
Update it, including the 5.10.63-v8+ kernel
Create a device tree from the running configuration (note: /dev/vcio does exist) with:
dtc -I fs -O dtb -o base.dtb /proc/device-tree
modify config.txt with device_tree=base.dtb
reboot
ls -l /dev/vcio

Expected behaviour
/dev/vcio is created

Actual behaviour
/dev/vcio is not created

System
Raspberry Pi 4 Model B Rev 1.2

Raspberry Pi reference 2021-05-07
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 071d3956353f2be964d2d337d3b123f1872b7f53, stage 2

Sep 28, 2021 11:27:35
Copyright (c) 2012 Broadcom
version 778b6a4f3c7d8d48bb63c02c47bcfbac79417bea (clean) (release) (start)

Linux raspberrypi 5.10.63-v8+ #1457 SMP PREEMPT Tue Sep 28 11:27:02 BST 2021 aarch64 GNU/Linux

Additional context
This same setup works as expected with the 5.10.63-v7 kernel and 32 bit RPiOS.

Note: The system info was typed not copied, so possible typos exist.

The text was updated successfully, but these errors were encountered:

0n0w1c · 2021-10-07T18:40:37Z

I believe the issue is here:
/linux/drivers/char/broadcom/vcio.c

np = of_find_compatible_node(NULL, NULL,
				     "raspberrypi,bcm2835-firmware");
if (!of_device_is_available(np))
		return -ENODEV;

vcio.fw = rpi_firmware_get(np);
if (!vcio.fw)
		return -ENODEV;

pelwell · 2021-10-07T21:51:55Z

Please upload the .dtb somewhere - Google Drive, etc. - so that I can download and analyse it.

0n0w1c · 2021-10-07T23:19:58Z

If you copy the base32.dtb to the 64-bit RPiOS, it will create /dev/vcio.

base32.dtb
base64.dtb

pelwell · 2021-10-08T09:30:45Z

I can reproduce the issue here - indeed, the rpi_firmware_get call is failing for reasons as yet undetermined.

pelwell · 2021-10-08T10:39:45Z

This appears to be a startup order problem, where the behaviour depends on the precise ordering of the nodes in the DTB. Apart from the fact that the UART is disabled in the 64-bit .dtb file, the ordering is the only difference.

In a successful start we see the mailbox being probed before the firmware, followed by vcio:

[    0.062026] rpi_firmware_init: ret 1 0
[    0.062086] rpi_firmware_init: ret 2 0
[    0.106753] bcm2835_mbox_probe:
[    0.106935] bcm2835_mbox_probe: 0
[    0.107820] rpi_firmware_probe(ffffff80408f3c00):
[    0.120649] rpi_firmware_probe(ffffff80408f3c00): OK
[    1.301066] vcio: vcio_init(/soc/firmware)
[    1.301097] vcio: A
[    1.301113] vcio: B

A failure case has the firmware probing before the mailbox, causing it to return -EPROBE_DEFER that will cause it to be probed again later. Unfortunately the vcio driver is initialised (there is no probe function - this isn't a platform driver) before the second, successful firmware probe, causing it to fail before it gets to point B in the init function. Return -EPROBE_DEFER doesn't help here - init functions don't appear to be retried.

[    0.062723] rpi_firmware_init: ret 1 0
[    0.062782] rpi_firmware_init: ret 2 0
[    0.109677] rpi_firmware_probe(ffffff80408f6400):
[    0.109726] rpi_firmware_probe(ffffff80408f6400): -> -517
[    0.110178] bcm2835_mbox_probe:
[    0.110347] bcm2835_mbox_probe: 0
[    1.307862] vcio: vcio_init(/soc/firmware)
[    1.307941] vcio: A
[    1.414695] rpi_firmware_probe(ffffff80408f6400):
[    1.415635] rpi_firmware_probe(ffffff80408f6400): OK

dtc has a sort option - -s - but unfortunately it makes both generated dtbs - base32 and base64 (*) - cause the vcio driver to fail.

I think we're fortunate not to have been hit by this problem sooner. In the longer term I think the vcio driver needs to be made a platform driver with a DT node under the firmware node - like firmware-clocks driver. That should sort out the ordering. In the short term, you should use base32.dtb - although you may want to regenerate it with the UART enabled.

(*) I'm using the names you gave the files, indicating how you generated them, but there is nothing 32-bit-specific about base32 and nothing 64-bit-specific about base64.

0n0w1c · 2021-10-09T01:46:57Z

Thank you for the detail, it was educational for me.

The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: raspberrypi#4620 Signed-off-by: Phil Elwell <[email protected]>

In order to resolve a potential startup order bug, the vcio driver has been rewritten as a platform driver that depends on a DT node for its instantiation and to locate the firmware driver. Add that DT node. See: raspberrypi#4620 Signed-off-by: Phil Elwell <[email protected]>

pelwell · 2021-10-12T09:41:49Z

See #4624 as a potential fix for the problem. It would be great if you could build your own kernel and confirm that it resolves the issue for you.

The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: raspberrypi#4620 Signed-off-by: Phil Elwell <[email protected]>

In order to resolve a potential startup order bug, the vcio driver has been rewritten as a platform driver that depends on a DT node for its instantiation and to locate the firmware driver. Add that DT node. See: raspberrypi#4620 Signed-off-by: Phil Elwell <[email protected]>

0n0w1c · 2021-10-13T21:14:28Z

Sorry for the delay, I will begin shortly.

0n0w1c · 2021-10-14T00:13:24Z

I compiled and installed, rebooted, recreated the .dtb, and rebooted... success!

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.63-v8 #1 SMP PREEMPT Wed Oct 13 17:31:02 CDT 2021 aarch64 GNU/Linux

pi@raspberrypi:~ $ ls -l /dev/vcio
crw------- 1 root root 10, 59 Oct 13 18:59 /dev/vcio

pi@raspberrypi:~ $ dmesg | grep -b3 -a3 vcio
12160-[    1.304507] bcm2835-aux-uart fe215040.serial: there is not valid maps for state default
12251-[    1.306699] iproc-rng200 fe104000.rng: hwrng registered
12310-[    1.307083] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
12404:[    1.307604] vcio soc:firmware:vcio: A
12445-[    1.307674] B
12462-[    1.308207] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000
12548-[    1.308631] cacheinfo: Unable to detect cache hierarchy for CPU 0
--
14700-[    1.409373] raspberrypi-firmware soc:firmware: Firmware hash is 778b6a4f3c7d8d48bb63c02c47bcfbac79417bea
14808-[    1.409921] uart-pl011 fe201000.serial: there is not valid maps for state default
14893-[    1.417404] bcm2835-aux-uart fe215040.serial: there is not valid maps for state default
14984:[    1.418041] vcio soc:firmware:vcio: A
15025-[    1.418112] B
15042-[    1.464334] pci 0000:00:00.0: enabling device (0000 -> 0002)
15106-[    1.464498] xhci_hcd 0000:01:00.0: xHCI Host Controller

The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: raspberrypi#4620 Signed-off-by: Phil Elwell <[email protected]>

In order to resolve a potential startup order bug, the vcio driver has been rewritten as a platform driver that depends on a DT node for its instantiation and to locate the firmware driver. Add that DT node. See: raspberrypi#4620 Signed-off-by: Phil Elwell <[email protected]>

pelwell · 2021-10-14T08:31:12Z

Thanks. I've removed those two stray lines of debug output.

I would have merged already except for an issue with device permissions. Changing the implementation to use the miscdevice framework also changes the SUBSYSTEM attribute, which is used by one of our udev rules to make the device accessible without sudo. Fortunately there is an easy fix - just change the rules from:

SUBSYSTEM=="vchiq", GROUP="video", MODE="0660"
SUBSYSTEM=="vc-sm", GROUP="video", MODE="0660"
SUBSYSTEM=="bcm2708_vcio", GROUP="video", MODE="0660"

to:

KERNEL=="vchiq", GROUP="video", MODE="0660"
SUBSYSTEM=="vc-sm", GROUP="video", MODE="0660"
KERNEL=="vcio", GROUP="video", MODE="0660"

You'll notice that the rule for vchiq has been modified in the same way because of an upcoming change in 5.16.

Our Bullseye releases will come with the new rules.

0n0w1c · 2021-10-14T11:45:44Z

Ah yes, in my hurried test I did forgot to change the udev rule to set the correct permissions.

Thank you for addressing this issue, well done.

graysky2 · 2021-10-14T13:43:34Z

@pelwell - In your post above, your from/to lines for "vc-sm" are identical. Was that your intention?

pelwell · 2021-10-14T13:47:46Z

Yes - the entry for vc-sm is unchanged.

The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: #4620 Signed-off-by: Phil Elwell <[email protected]>

In order to resolve a potential startup order bug, the vcio driver has been rewritten as a platform driver that depends on a DT node for its instantiation and to locate the firmware driver. Add that DT node. See: #4620 Signed-off-by: Phil Elwell <[email protected]>

0n0w1c · 2021-10-16T00:10:40Z

To bring this full circle:
5.10.73-2-MANJARO-ARM
crw-rw---- 1 root video 10, 59 Oct 14 16:33 /dev/vcio
Successfully booted via uefi, grub and a custom .dtb

The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: #4620 Signed-off-by: Phil Elwell <[email protected]>

In order to resolve a potential startup order bug, the vcio driver has been rewritten as a platform driver that depends on a DT node for its instantiation and to locate the firmware driver. Add that DT node. See: #4620 Signed-off-by: Phil Elwell <[email protected]>

The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: #4620 Signed-off-by: Phil Elwell <[email protected]>

In order to resolve a potential startup order bug, the vcio driver has been rewritten as a platform driver that depends on a DT node for its instantiation and to locate the firmware driver. Add that DT node. See: #4620 Signed-off-by: Phil Elwell <[email protected]>

The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: #4620 Signed-off-by: Phil Elwell <[email protected]>

Add module for accessing the mailbox property channel through /dev/vcio. Was previously in bcm2708-vcio. Signed-off-by: Noralf Trønnes <[email protected]> char: vcio: Add compat ioctl handling There was no compat ioctl handler, so 32 bit userspace on a 64 bit kernel failed as IOCTL_MBOX_PROPERTY used the size of char*. Signed-off-by: Dave Stevenson <[email protected]> char: vcio: Fail probe if rpi_firmware is not found. Device Tree is now the only supported config mechanism, therefore uncomment the block of code that fails the probe if the firmware node can't be found. Signed-off-by: Dave Stevenson <[email protected]> drivers: char: vcio: Use common compat header The definition of compat_ptr is now common for most platforms, but requires the inclusion of <linux/compat.h>. Signed-off-by: Phil Elwell <[email protected]> char: vcio: Rewrite as a firmware node child The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: #4620 Signed-off-by: Phil Elwell <[email protected]>

Add module for accessing the mailbox property channel through /dev/vcio. Was previously in bcm2708-vcio. Signed-off-by: Noralf Trønnes <[email protected]> char: vcio: Add compat ioctl handling There was no compat ioctl handler, so 32 bit userspace on a 64 bit kernel failed as IOCTL_MBOX_PROPERTY used the size of char*. Signed-off-by: Dave Stevenson <[email protected]> char: vcio: Fail probe if rpi_firmware is not found. Device Tree is now the only supported config mechanism, therefore uncomment the block of code that fails the probe if the firmware node can't be found. Signed-off-by: Dave Stevenson <[email protected]> drivers: char: vcio: Use common compat header The definition of compat_ptr is now common for most platforms, but requires the inclusion of <linux/compat.h>. Signed-off-by: Phil Elwell <[email protected]> char: vcio: Rewrite as a firmware node child The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: raspberrypi/linux#4620 Signed-off-by: Phil Elwell <[email protected]>

Add module for accessing the mailbox property channel through /dev/vcio. Was previously in bcm2708-vcio. Signed-off-by: Noralf Trønnes <[email protected]> char: vcio: Add compat ioctl handling There was no compat ioctl handler, so 32 bit userspace on a 64 bit kernel failed as IOCTL_MBOX_PROPERTY used the size of char*. Signed-off-by: Dave Stevenson <[email protected]> char: vcio: Fail probe if rpi_firmware is not found. Device Tree is now the only supported config mechanism, therefore uncomment the block of code that fails the probe if the firmware node can't be found. Signed-off-by: Dave Stevenson <[email protected]> drivers: char: vcio: Use common compat header The definition of compat_ptr is now common for most platforms, but requires the inclusion of <linux/compat.h>. Signed-off-by: Phil Elwell <[email protected]> char: vcio: Rewrite as a firmware node child The old vcio driver is a simple character device that manually locates the firmware driver. Initialising it before the firmware driver causes a failure, and no retries are attempted. Rewrite vcio as a platform driver that depends on a DT node for its instantiation and the location of the firmware driver, making use of the miscdevice framework to reduce the code size. N.B. Using miscdevice changes the udev SUBSYSTEM string, so a change to the companion udev rule is required in order to continue to set the correct device permissions, e.g.: KERNEL="vcio", GROUP="video", MODE="0660" See: #4620 Signed-off-by: Phil Elwell <[email protected]>

0n0w1c mentioned this issue Oct 7, 2021

/dev/vcio not created when using custom dtb raspberrypi/firmware#1634

Closed

pelwell mentioned this issue Oct 12, 2021

vcio rewrite #4624

Merged

pelwell closed this as completed Oct 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

/dev/vcio is not created when using a custom dtb on 64 bit RPiOS #4620

/dev/vcio is not created when using a custom dtb on 64 bit RPiOS #4620

0n0w1c commented Oct 7, 2021 •

edited

Loading

0n0w1c commented Oct 7, 2021 •

edited

Loading

Uh oh!

pelwell commented Oct 7, 2021

Uh oh!

0n0w1c commented Oct 7, 2021

Uh oh!

pelwell commented Oct 8, 2021 •

edited

Loading

Uh oh!

pelwell commented Oct 8, 2021 •

edited

Loading

Uh oh!

0n0w1c commented Oct 9, 2021

Uh oh!

pelwell commented Oct 12, 2021

Uh oh!

0n0w1c commented Oct 13, 2021

Uh oh!

0n0w1c commented Oct 14, 2021

Uh oh!

pelwell commented Oct 14, 2021

Uh oh!

0n0w1c commented Oct 14, 2021

Uh oh!

graysky2 commented Oct 14, 2021

Uh oh!

pelwell commented Oct 14, 2021

Uh oh!

0n0w1c commented Oct 16, 2021

Uh oh!

/dev/vcio is not created when using a custom dtb on 64 bit RPiOS #4620

/dev/vcio is not created when using a custom dtb on 64 bit RPiOS #4620

Comments

0n0w1c commented Oct 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

0n0w1c commented Oct 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelwell commented Oct 7, 2021

Uh oh!

0n0w1c commented Oct 7, 2021

Uh oh!

pelwell commented Oct 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelwell commented Oct 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0n0w1c commented Oct 9, 2021

Uh oh!

pelwell commented Oct 12, 2021

Uh oh!

0n0w1c commented Oct 13, 2021

Uh oh!

0n0w1c commented Oct 14, 2021

Uh oh!

pelwell commented Oct 14, 2021

Uh oh!

0n0w1c commented Oct 14, 2021

Uh oh!

graysky2 commented Oct 14, 2021

Uh oh!

pelwell commented Oct 14, 2021

Uh oh!

0n0w1c commented Oct 16, 2021

Uh oh!

0n0w1c commented Oct 7, 2021 •

edited

Loading

0n0w1c commented Oct 7, 2021 •

edited

Loading

pelwell commented Oct 8, 2021 •

edited

Loading

pelwell commented Oct 8, 2021 •

edited

Loading