oversized secure-boot ramdisk silently truncated

[cleverca22]

Mandatory information

• Raspberry Pi model: pi4b

Describe the bug
if the boot.img file is over 64mb, it will silently be truncated to 64mb, and then fail the boot.sig validation

To Reproduce
create a properly signed boot.img that is 100mb in size

Expected behaviour
at a bare minimum, report that it is too big on the uart logs, so other users dont have to decompile just to find the size limit
the size limit should also be documented and checked in the scripts for building the image, they currently warn for >20mb, but dont tell you that >64mb wont work

ideally, have a more dynamic buffer size, but i can see that being difficult

Bootloader version and configuration
RPi: BOOTLOADER release VERSION:a6afaeaa DATE: Oct 5 2021 TIME: 08:53:57 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1633420437 0xd328cdce 0x00c03112 0x0006f72c

[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=192.168.2.15
TFTP_PREFIX=0
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
ENABLE_SELF_UPDATE=1
SIGNED_BOOT=1

gpio=21=ip,pu

[gpio21=0]
BOOT_ORDER=0x3

[gpio21=1]
BOOT_ORDER=0x5421

[none]
FREEZE_VERSION=0

Additional context
Add any other context about the problem here.

RPi: BOOTLOADER release VERSION:a6afaeaa DATE: Oct  5 2021 TIME: 08:53:57 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1633420437 0xd328cdce 0x00c03112 0x0006f72c
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Micron' 16Gb x2 total-size: 32 Gbit 3200

XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 811
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Reset USB port-power 1000 ms
xhci_set_port_power 1 0
xhci_set_port_power 2 0
xhci_set_port_power 3 0
xhci_set_port_power 4 0
xhci_set_port_power 5 0
xhci_set_port_power 1 1
xhci_set_port_power 2 1
xhci_set_port_power 3 1
xhci_set_port_power 4 1
xhci_set_port_power 5 1
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 18
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 19
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Boot mode: SD (01) order 542
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
OCR 80ff8000 [2]
CID: 0002544d5341303247092b62d0be00c6
CSD: 002e00325b5aa3b4ffffff800a800000
SD: bus-width: 4 spec: 2 SCR: 0x02258000 0x01000000
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 12500000 Hz actual: 12500000 HZ div: 20 (10) status: 0x1fff0000 delay: 8
MBR: 0x00000800, 1046528 type: 0x0b
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 16 lba: 2048 oem: 'mkfs.fat' volume: '  V       ^ '
rsc 16 fat-sectors 256 c-count 65373 c-size 16
root dir cluster 1 sectors 32 entries 512
FAT16 clusters 65373
secure-boot
Loading boot.img ...
SIG boot.sig db8002deb34d482a4cb46dd6cb25a7904b90aa6c4bbaf054f288268021292600 1634538695
Verifying
Bad signature boot.img
Error 12 loading boot.img
Boot mode: NETWORK (02) order 54

yes, the signature is valid, it fails with a 100mb image, but works with a 64mb image, signed using the same directions and key

[timg236]

Thanks. Yes, there should be an explicit error message for this so we'll add that.

The current limit as you say is 64MB, maybe 128MB could be supported although the signature check will be a bit slow. There's some wiggle room in the memory map but I don't want to commit to things that will cause headaches in the future until the use-cases are better understood.

[cleverca22]

for context, my initrd is 50mb
i had to manually gzip -9 the linux kernel, and delete all unused start*.elf files, just to make it barely fit within 64mb

but i can see 2 ways i could optimize my end
either shrink the initrd more
or put the kernel+initrd outside of the firmware ramdisk, so the firmware only validates uboot, and uboot validates linux+initrd

[timg236]

The latest beta-release (also in usbboot) should help here. The limit is increased to 96MB and some of the error messages have been improved
978ced9 and
raspberrypi/usbboot@80945ed

[ibot3]

Thank you for increasing the limit.

I am using unsigned/non-secure images and really had problems with the 64MB limit.
96MB should be enough for now, but is there a reason why it can't increased further for images without signature?

[ibot3]

Hmm, @timg236 it seems that the new limit somehow doesn't apply..

I still says this in dmesg and just truncates the image after 64MB:

rootfs image is not initramfs (XZ-compressed data is corrupt): looks like an initrd
Freeing initrd memory: 65532K

The initrd file has around ~68MB.

[cleverca22]

@ibot3 this ticket is about the firmware ramdisk, but your error message looks like a linux ramdisk, which doesnt have a size limit

when the firmware ramdisk gets truncated, that also changes the hash, causing the signature to be invalid, so it wont even boot

[ibot3]

Okay, sorry. Then I just misunderstood something.

However, there seems to be a limit of 64MB which is really annoying:
https://forums.raspberrypi.com/viewtopic.php?t=269993#p1642596

But I don't know where this limit is applied. But I should probably create an other issue for this (I just dk where).

[timg236]

It's 96MB and it's applied by the bootloader - the logging should indicate this more clearly.

That's still pretty big for a ramdisk that should only include kernel, start.elf and minimal initrd for secure-boot

[ibot3]

I am using the initrd to store a minimal bootstrapped debian in it which is around 300MB uncompressed.

It's working fine as long as the initrd is below 64MB. But if that's not the case, the initrd seems to get truncated which causes a corrupted system, because important files are missing.

I am using netboot for this by the way.

-rw-r--r--  1 root root  19K Oct 29 11:52 COPYING.linux
-rw-r--r--  1 root root 1.6K Oct 29 11:52 LICENCE.broadcom
-rw-r--r--  1 root root    0 Nov 29 11:41 UART
-rw-r--r--  1 root root  27K Nov 29 11:56 bcm2710-rpi-2-b.dtb
-rw-r--r--  1 root root  29K Nov 29 11:56 bcm2710-rpi-3-b-plus.dtb
-rw-r--r--  1 root root  29K Nov 29 11:56 bcm2710-rpi-3-b.dtb
-rw-r--r--  1 root root  27K Nov 29 11:56 bcm2710-rpi-cm3.dtb
-rw-r--r--  1 root root  49K Nov 29 11:56 bcm2711-rpi-4-b.dtb
-rw-r--r--  1 root root  49K Nov 29 11:56 bcm2711-rpi-400.dtb
-rw-r--r--  1 root root  50K Nov 29 11:56 bcm2711-rpi-cm4.dtb
-rw-r--r--  1 root root  20K Nov 29 11:56 bcm2837-rpi-3-a-plus.dtb
-rw-r--r--  1 root root  21K Nov 29 11:56 bcm2837-rpi-3-b-plus.dtb
-rw-r--r--  1 root root  21K Nov 29 11:56 bcm2837-rpi-3-b.dtb
-rw-r--r--  1 root root  20K Nov 29 11:56 bcm2837-rpi-cm3-io3.dtb
-rw-r--r--  1 root root  52K Oct 29 11:52 bootcode.bin
-rw-r--r--  1 root root   99 Nov 29 11:41 cmdline.txt
-rw-r--r--  1 root root  279 Nov 29 11:41 config.txt
-rw-r--r--  1 root root 7.2K Oct 29 11:52 fixup.dat
-rw-r--r--  1 root root 5.3K Oct 29 11:52 fixup4.dat
-rw-r--r--  1 root root 3.2K Oct 29 11:52 fixup4cd.dat
-rw-r--r--  1 root root 8.3K Oct 29 11:52 fixup4db.dat
-rw-r--r--  1 root root 8.3K Oct 29 11:52 fixup4x.dat
-rw-r--r--  1 root root 3.2K Oct 29 11:52 fixup_cd.dat
-rw-r--r--  1 root root  11K Oct 29 11:52 fixup_db.dat
-rw-r--r--  1 root root  11K Oct 29 11:52 fixup_x.dat
-rw-r--r--  1 root root  68M Nov 29 12:06 initrd
-rw-r--r--  1 root root  20M Nov 29 11:56 kernel8.img
-rwxr-xr-x  1 root root 429K Nov 29 11:41 msd.elf
drwxr-xr-x  2 root root  16K Nov 29 11:56 overlays
-rw-r--r--  1 root root 2.9M Oct 29 11:52 start.elf
-rw-r--r--  1 root root 2.2M Oct 29 11:52 start4.elf
-rw-r--r--  1 root root 783K Oct 29 11:52 start4cd.elf
-rw-r--r--  1 root root 3.6M Oct 29 11:52 start4db.elf
-rw-r--r--  1 root root 2.9M Oct 29 11:52 start4x.elf
-rw-r--r--  1 root root 783K Oct 29 11:52 start_cd.elf
-rw-r--r--  1 root root 4.6M Oct 29 11:52 start_db.elf
-rw-r--r--  1 root root 3.6M Oct 29 11:52 start_x.elf
``

[timg236]

There's no benefit in including all the start.elf variants or bootcode.bin.

N.B. The purpose of the ramdisk is just to bootstrap secure-boot and not provide alternate methods for running Linux from memory e.g. you could use NFS to load the root filesystem

[ibot3]

Okay I found a guide which uses an cow-overlayfs with an ro NFS: https://blockdev.io/read-only-rpi/

But imo the initrd solution is more clean. And with an NFS, I have a dependency on the availability of the NFS server.
I would like to run the pis independent of the network.

We have 30+ PoE powered pis and I just want to be able to create a new image and just restart all pis to keep them up to date.

So would there be a way to increase this limit of 64MB?

[timg236]

If you use the latest beta EEPROM 2021-11-22 then the limit is 96MB.
https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/beta/pieeprom-2021-11-22.bin

I think in the future we might be able to go as high as 256MB assuming you can tolerate the load time. However, there's a few memory map things that we'd need to think through first e..g. conflicting with large amounts of GPU memory and random Kernel load addresses i.e. not a very short term fix

[timg236]

Closing since this the limit was increased to 96MB in the stable release. Larger RAM disks might be supported in future but will probably have the caveat that GPU memory will be automatically reduced instead. That should be ok because KMS + libcamera mostly remove the need for GPU memory.

[ibot3]

@timg236
The limit has been raised, but I still have the same problems with 2022-08-02 (and also all other versions):

rootfs image is not initramfs (XZ-compressed data is corrupt): looks like an initrd
Freeing initrd memory: 65532K

So there still seems to be some limit of 64MB and I don't understand where it comes from.

However I am using initramfs initrd in config.txt instead of boot_ramdisk.