bootcode.bin: Initrd loading seems to be broken #1750

flipreverse · 2022-09-22T20:06:56Z

Describe the bug
bootcode.bin is responsible for loading the kernel image as well as the initrd.
This, however, does not happen consistently. If the boot uart is enabled, it works.
Otherwise, not.

To reproduce
Create an initrd mkinitramfs -o /boot/init.gz for my overlayfs setup.
Enable it in /boot/config.txt: initramfs init.gz
Ensure boot uart is disabled.

Expected behaviour
The initrd is loaded, and my overlayfs is set up properly.

Actual behaviour
The kernel fails to load the initrd. See dmesg output below.

System

Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
Raspberry Pi 2 Model B Rev 1.1
Which OS and version (cat /etc/rpi-issue)?
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 66255495f29be5d09b765d081aff6fc0f11e59b4, stage2
Which firmware version (vcgencmd version)?

Aug 26 2022 14:05:32 
Copyright (c) 2012 Broadcom
version 102f1e848393c2112206fadffaaf86db04e98326 (clean) (release) (start_x)

Which kernel version (uname -a)?
Linux hyperion 5.15.68-v7+ #1587 SMP Tue Sep 20 11:15:23 BST 2022 armv7l GNU/Linux

Logs

[  +0.000757] Trying to unpack rootfs image as initramfs...
[  +0.000262] rootfs image is not initramfs (invalid magic at start of compressed archive); looks like an initrd

Additional context
If I, however, enable the boot uart as described here, everything works fine. The initrd is loaded, and my overlayfs works. I successfully repeated this procedure multiple times, when writing this post.

The text was updated successfully, but these errors were encountered:

timg236 · 2022-09-23T08:40:05Z

bootcode.bin only loads start.elf and fixup.dat. The OS and initrd are loaded by start.elf
What's the latest known good version of start.elf that works for you (vcgencmd version)?

UART logging adds some delays. If you revert the BOOT_UART change and just add uart_2ndstage=1 to config.txt does that have the same effect?

flipreverse · 2022-09-23T12:02:24Z

Adding uart_2ndstage=1to /boot/config.txt seems to have the same effect.
Which files do I have to replace when moving backwards? Just start_x.elf and fixup.dat?

popcornmix · 2022-09-23T12:06:48Z

Which files do I have to replace when moving backwards? Just start_x.elf and fixup.dat?

Make sure they are matching (e.g. start_x.elf and fixup_x.dat) and they are the ones you are using (start_x.elf only used if start_x=1 is in config.txt). But should be easy to test if you are doing the right thing (it will boot and vcgencmd version will change output to match).

flipreverse · 2022-09-23T14:20:44Z

I think I narrowed it down to these commits:
Last working commit in this repo is: 329af8a59d91ea023ce3e2566e2ccd4ec0624438

Jun 15 2022 20:08:33
Copyright (c) 2012 Broadcom
version ede5f7d365ef42ebbd4aa144ee5b51ea75604c82 (clean) (release) (start_x)

Next commit df569e043fe498afac2506ed45765d58aa80c408 seems to broken:

Jul  4 2022 14:42:50
Copyright (c) 2012 Broadcom
version 01db5c05be1b3594c8f040ed7c1ac2748890c2c8 (clean) (release) (start_x)

pelwell · 2022-09-23T14:35:22Z

Thanks - that's really helpful. Unfortunately for us that short list of commits consists of 3 that are clocking-related, and 5 which look as though they should be harmless.

flipreverse · 2022-09-23T14:40:41Z

Yikes. If you have any new firmware binaries, feel free to contact me. :)

pelwell · 2022-09-23T15:27:32Z

There is a trial firmware, with just the 3 clock-related patches reverted, available to download here: https://drive.google.com/file/d/1g-F3KC2Kp14MSrJR7gYPPCQ_eK43TyEv/view?usp=sharing
It boots for me on a 2B, but I've not tested it beyond that.

flipreverse · 2022-09-23T18:32:15Z

I previously used the _x files. Does this matter?
If so, the google drive offers wrong files.

popcornmix · 2022-09-23T18:40:53Z

Remove start_x=1 from config.txt and use the files start.elf/fixup.dat.

flipreverse · 2022-09-23T20:42:05Z

Just to be sure. vcgencmd version reports:

Sep 23 2022 15:58:05 
Copyright (c) 2012 Broadcom
version d3dbb0500a7fdd0dec3183a56f25238cb45e19b5 (tainted) (release) (start)

However, that didn't do the trick. :( I still have to add uart_2ndstage=1 to my config.

pelwell · 2022-09-26T15:24:16Z

Let's try again, this time reverting all the firmware commits between those two hashes: https://drive.google.com/file/d/1e0jiW91R8SD7l-rZRaCqmEv4A9utZ64e/view?usp=sharing
This time I've supplied the x variants, so you should be able to restore start_x=1.

flipreverse · 2022-09-26T15:35:16Z

Sep 26 2022 15:34:42 
Copyright (c) 2012 Broadcom
version ad2c2750f3ec6fd0aa84bb6d931d9f793a84b7c8 (tainted) (release) (start_x)

[all]
initramfs init.gz
#uart_2ndstage=1
start_x=1

This one works.

pelwell · 2022-09-26T15:39:37Z

That's a relief - I don't like Heisenbugs. If I can't see the problem after reading the remaining commits, I may have to resort to a binary chop.

pelwell · 2022-09-26T16:27:47Z

Next one, this time with just one specific commit reverted: https://drive.google.com/file/d/1KX5VtdS5ItX5_HPHDyObhR-4k8iNefvY/view?usp=sharing

pelwell · 2022-09-26T16:50:12Z

Could you also tell me if you have any cameras attached, and if so, which ones?

flipreverse · 2022-09-26T17:19:10Z

Could you also tell me if you have any cameras attached, and if so, which ones?

Yes, I do have one attached. I think it's this one: 'Raspberry Pi Camera Module 2 NoIR'.

It works with:

Sep 26 2022 17:09:28 
Copyright (c) 2012 Broadcom
version 4bcce7924dc536394dec4b54d5fc2fae196d2394 (tainted) (release) (start_x)

pelwell · 2022-09-26T19:13:52Z

Ooh, interesting. Sorry, @naushir - that's a top-of-tree firmware + a reversion of 89af8e130aa635d8c31c0c3394461a7605858260.

naushir · 2022-09-27T07:57:18Z

Oops! I have no idea how that change could cause this issue. I'll look into it....

naushir · 2022-09-28T12:28:49Z

@flipreverse can you tell me the size of your init.gz file? Additionally, if you unplug your camera, do you still get the failure on boot?

flipreverse · 2022-09-28T16:33:44Z

@flipreverse can you tell me the size of your init.gz file? Additionally, if you unplug your camera, do you still get the failure on boot?

The init.gz is about 8.3MB.

Yes, it works without activating the secondary stage uart if I unplug my camera. Firmware version:

Aug 26 2022 14:05:32 
Copyright (c) 2012 Broadcom
version 102f1e848393c2112206fadffaaf86db04e98326 (clean) (release) (start_x)

…ning block firmware: arm_loader: Raise maximum gzipped kernel size firmware: arm-loader: Indicate tryboot status via /proc/device-tree/chosen/bootloader/tryboot firmware: arm_loader: Increase TFTP block size to 1468 bytes See: raspberrypi/rpi-eeprom#375 firmware: Add kernel= logging firmware: camera_auto_detect changes See: #1750

…ning block firmware: arm_loader: Raise maximum gzipped kernel size firmware: arm-loader: Indicate tryboot status via /proc/device-tree/chosen/bootloader/tryboot firmware: arm_loader: Increase TFTP block size to 1468 bytes See: raspberrypi/rpi-eeprom#375 firmware: Add kernel= logging firmware: camera_auto_detect changes See: raspberrypi/firmware#1750

popcornmix · 2022-09-30T13:37:44Z

@flipreverse there is an rpi-update firmware with a possible fix for this.

flipreverse · 2022-10-03T17:02:03Z

@flipreverse there is an rpi-update firmware with a possible fix for this.

Sep 30 2022 14:28:00 
Copyright (c) 2012 Broadcom
version afe2b3f5f60315137568e903508e7eae8b6543a4 (clean) (release) (start_x)

This version looks good.

Out of curiosity: What was the cause of this bug?

popcornmix · 2022-10-03T17:14:01Z

@naushir provided the fix, so I'm not certain.
The issue seemed to start with putting camera detection into a separate gpu thread.
The fix was a partial revert of that.

naushir · 2022-10-04T13:04:25Z

Yes, it reverted the change that moved the auto-detection routine into a separate thread.

timg236 · 2022-11-28T15:47:55Z

Fix was released in 2b3cef2

timg236 closed this as completed Nov 28, 2022

bootcode.bin: Initrd loading seems to be broken #1750

bootcode.bin: Initrd loading seems to be broken #1750

Comments

flipreverse commented Sep 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

timg236 commented Sep 23, 2022

Uh oh!

flipreverse commented Sep 23, 2022

Uh oh!

popcornmix commented Sep 23, 2022

Uh oh!

flipreverse commented Sep 23, 2022

Uh oh!

pelwell commented Sep 23, 2022

Uh oh!

flipreverse commented Sep 23, 2022

Uh oh!

pelwell commented Sep 23, 2022

Uh oh!

flipreverse commented Sep 23, 2022

Uh oh!

popcornmix commented Sep 23, 2022

Uh oh!

flipreverse commented Sep 23, 2022

Uh oh!

pelwell commented Sep 26, 2022

Uh oh!

flipreverse commented Sep 26, 2022

Uh oh!

pelwell commented Sep 26, 2022

Uh oh!

pelwell commented Sep 26, 2022

Uh oh!

pelwell commented Sep 26, 2022

Uh oh!

flipreverse commented Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelwell commented Sep 26, 2022

Uh oh!

naushir commented Sep 27, 2022

Uh oh!

naushir commented Sep 28, 2022

Uh oh!

flipreverse commented Sep 28, 2022

Uh oh!

popcornmix commented Sep 30, 2022

Uh oh!

flipreverse commented Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

popcornmix commented Oct 3, 2022

Uh oh!

naushir commented Oct 4, 2022

Uh oh!

timg236 commented Nov 28, 2022

Uh oh!

flipreverse commented Sep 22, 2022 •

edited

Loading

flipreverse commented Sep 26, 2022 •

edited

Loading

flipreverse commented Oct 3, 2022 •

edited

Loading