-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Regression booting rpi3/4 with U-Boot on recent firmware #1619
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Confirming this issue... SD card throughput plummets with recent firmware. From u-boot 2021.07 with rpi_arm64_defconfig:
Currently:
Previously:
the current 1.20210831 tag is affected, 1.20210805 is not. 25e2b59 seems to introduce the issue. 69cfd9f is not affected. Works:
affected:
|
This is hitting the latest edk2 builds as well. I debugged it a bit in uboot last week and the mailbox is returning a 0 clock rate. I need to see if the mailbox structure changed/whatever but I'm away from the pi's today. |
Any further information you can provide about the clock in question and the circumstances surrounding the 0 rate would be appreciated. |
I put some print statements in u-boot:
on previous firmware:
So on newer firmware is returning 0 for clock BCM2835_MBOX_CLOCK_ID_CORE (4) |
Ok, reproduced it fully with a rpi4-8G model in edk2. When it works (with a newer sd card) ... But that last bit when it fails: RpiFirmwareGetClockRate: Get Clock Rate: Tag=30002 ClockId=C The fedora defect: https://bugzilla.redhat.com/show_bug.cgi?id=1999180 Notes that uboot is splatting: sdhci_setup_cfg: Hardware doesn't specify base clock frequency which happens the same way (the mailbox is returning that the clock rate is 0) but it also notes that its dependent on SD card, which seems to be the case since I spent an hour or so thinking I was crazy because I couldn't get it to trigger today. Further, its not apparently some kind of powersave/whatever going on because If I issue a request to set the clockrate to 100mhz on the emmc2 right before requesting the current state it will accept the request, but then immediately return that the clockrate is 0. |
Please note, that I am seeing the same behaviour here with a RaspberryPi2 and the latest 1.20210831 firmware. As soon as I use that version rather than the previous 1.20210805 one the SD card performance within u-boot drops significantly (from ~ 23MB/s to 120KB/s). So it seems that this is not limited to a Pi3 or Pi4 but also affects the older Pi2 as well. |
version breaks fast sd card use in u-boot environments due to invalid base freq spec. This fixes #1405 and refs raspberrypi/firmware#1619
Is there any further information needed to help with this issue? |
@popcornmix This seems to be a direct consequence of making |
The fix for this was to make If uboot is just using this call to get a fixed clock without setting it, it will now get zero returned. It may be better for uboot to call The other option is to use |
@popcornmix thanks I thought @jlinton mentioned that setting the clock and then retrieving it still returned 0... I can try playing around with uboot to see what happens with:
|
I'd expect setting clock and retrieving it to give the value you set. Let me now if you can confirm that. |
MIN clock, or MAX clock? I think in both uboot/edk2 the returned clock rate is being used to select the smallest possible divisor for the given card (aka run it as fast as possible). I don't think either of those project is really interested in a dynamic clock value, but rather its peak. If the actual clock rate is lower, AFAIK SD cards should be able to deal with that. The problem is if it ends up going over the card max. |
MAX clock may be safer if you don't want a divided down value to exceed a specced value. |
I can confirm that, within u-boot at least, if i issue
Whereas without the
uboot will boot fine with either |
What happens if MAX is read, and then set? Does it actually adjust the clock rate near the max? I would imagine that would be the safe method of maintaining the higher speeds? (I will probably take another pass with edk2 later this evening) |
@jlinton i'll test that in a bit. I was trying to divine what's being done in the Linux driver, the logic is a little different than what's being done in uboot https://github.com/raspberrypi/linux/blob/1.20210831/drivers/mmc/host/bcm2835-sdhost.c#L2127 |
Setting the core clock (4) to the max maintains the higher speeds
though i'm a little wary of changing the core clock to max (core is currently being queried due to this note #953 (comment)) |
So, i'm looking at the code now, and I now remember the dance edk2+ACPI is playing with Linux. (and I think this is true of DT too for the EMMC2). Linux, in the case of the EMMC2, is depending on the base clock rate to actually reflect what is in the SDHCI base clock frequency field (which is 100Mhz). In the case of the older EMMC which doesn't provide a meaningful cap's register, we are using the generic linux sdhci-caps/mask property to override the field to 250mhz. So, we can play this dance and change the freq but we need to override the sdhci base frequency fields being fed into Linux. Leading me to think, I'm just going to validate that the requested frequency is between min->max and request 100 (or 250) mhz. But, I'm not sure if I'm doing something wrong (I don't think so, we can override the core clock/etc) but I've tried changing the emmc2 clock (ClockId=C) with both the skip turbo and !skip turbo set, and I read it immediately back and its still zero, but only with the card that's failing. Otherwise, everything is fine (newer cards). The card that consistently fails for me is an old patriot 8GB SDHC class 4 card (SD2.0). This is following a 1.5Ghz request of the core clock, which is responding that the clock rate is what we requested. Further, if I ignore the clock=0 return, assume the clock is what we requested, everything seem to work fine, including Linux. |
I assumed you were talking about something internal to the vc firmware since I don't see that here: https://github.com/raspberrypi/firmware/wiki/Mailbox-property-interface#get-clock-rate or is there a change not documented there? |
@vfazio With a supporting firmware (which they all will have been since the Pi 3 was launched) the sdhost driver delegates the SD clock management to the firmware, relying on it to adjust the divisor as the core clock changes. @jlinton Yes, I was referring to a change in the firmware code that implements GET_CLOCK_RATE. |
Just wanted to add that we are seeing the same results reported above with our bare metal kernels. Any firmware version from commit 25e2b59 always returns 0 for
regardless of whether These all return correct values for Other clocks such as Commit 57e7fd1 in March 2021 changed the response for Clearly the sequence of calling |
Did you have a patch published somewhere to use as a testing base? |
@nullr0ute I'll push something soonish. There may be two patches, one to implement the logic that's being used in Linux re letting the firmware manage the clock, and another likely to have some safeguard behavior if I need to test that second patch w/o the first patch to make sure it works as intended. |
In rpi-firmware 25e2b597ebfb2495eab4816a276758dcc6ea21f1 and later, the GET_CLOCK_RATE mailbox property was changed to return the last value set by SET_CLOCK_RATE. raspberrypi/firmware#1619 (comment) U-Boot does not call SET_CLOCK_RATE so bcm2835_get_mmc_clock will return zero and can cause degraded MMC performance. Calling SET_CLOCK_RATE fixes the frequency of the clock to a specific value and disables the firmware's clock scaling so is not an option. Instead, fallback to GET_MAX_CLOCK_RATE in bcm2835_get_mmc_clock if GET_CLOCK_RATE returns zero. Signed-off-by: Vincent Fazio <[email protected]>
@nullr0ute This is based on the 2021.07 release: https://github.com/vfazio/u-boot/commits/vfazio-rpi-mmc-clock-fix but they'll probably apply cleanly to 2021.04. I'll try to submit the series to upstream tomorrow once i'm comfortable with the commit message wording. |
In rpi-firmware 25e2b597ebfb2495eab4816a276758dcc6ea21f1, the GET_CLOCK_RATE mailbox property was changed to return the last value set by SET_CLOCK_RATE. raspberrypi/firmware#1619 (comment) Due to this change in firmware behavior, bcm2835_get_mmc_clock now returns a clock rate of zero since we do not issue SET_CLOCK_RATE. This results in degraded MMC performance. SET_CLOCK_RATE fixes the clock to a specific value and disables scaling so is not an ideal solution. Instead, fallback to GET_MAX_CLOCK_RATE in bcm2835_get_mmc_clock if GET_CLOCK_RATE returns zero. Signed-off-by: Vincent Fazio <[email protected]>
In rpi-firmware 25e2b597ebfb2495eab4816a276758dcc6ea21f1, the GET_CLOCK_RATE mailbox property was changed to return the last value set by SET_CLOCK_RATE. raspberrypi/firmware#1619 (comment) Due to this change in firmware behavior, bcm2835_get_mmc_clock now returns a clock rate of zero since we do not issue SET_CLOCK_RATE. This results in degraded MMC performance. SET_CLOCK_RATE fixes the clock to a specific value and disables scaling so is not an ideal solution. Instead, fallback to GET_MAX_CLOCK_RATE in bcm2835_get_mmc_clock if GET_CLOCK_RATE returns zero. Signed-off-by: Vincent Fazio <[email protected]>
The clock will be set to the maximum of all requests. Also there may be other constraints, such as clocks which share a PLL. You'd probably have to give an example of the clock and value set, and whether the outcome was as expected. |
@popcornmix what is the expected behavior of
I've just retested with commit f9bc224 and for Other clocks such as CORE and CPU still return the value that was set. We've added an additional check so that if We are aware of other projects seeing the same results as well. |
It returns the value requested by the arm.
let me check what that is doing. |
I can reproduce. For those clocks it's not behaving as designed. I'll have a think about best way to fix it. |
BTW, the patch is not backward-compatible: firmware from 03.03.2021 doesn't work with U-Boot patched with this patch.
|
Does it complain if the second patch isn't applied? |
... and does not work with It seems to me that it might be better approach just to copy all clocks initialisation from old start.elf into U-Boot code... |
Unfortunately I do not have an rpi4 to test on. I tested with a CM3 with the rpi_arm64_defconfig and didn't have any issues, so if anyone is able to assist with debug efforts I'd appreciate it. |
Are you able to test this firmware and report if it behaves as you'd like. |
@popcornmix who is the firmware bundle targeting? Is this wrt #1619 (comment) ? |
The firmware is for any users experiencing "Regression booting rpi3/4 with U-Boot on recent firmware". |
@popcornmix works on my CM3 in a quick test. Looks like GET_CLOCK_RATE now returns a value (u-boot built without my patches on top)
|
@popcornmix, Tested and confirmed that Tested on Pi 3B+ and Pi 4B, results from Pi 4B. Release firmware:
Test firmware:
Seems to be resolved, thank you. |
See: #1619 firmware: dispmanx: Demote null eptr from vcos_verify to no warning See: raspberrypi/linux#4592 firmware: filesystem: sdcard: Probe FAT type in GPT ESD partitions
See: raspberrypi/firmware#1619 firmware: dispmanx: Demote null eptr from vcos_verify to no warning See: raspberrypi/linux#4592 firmware: filesystem: sdcard: Probe FAT type in GPT ESD partitions
Fix should be in latest rpi-update firmware. |
We primarily use tagged versions of the rpi firmware. Is there a tag scheduled at some point? |
At some point, but I can't say when exactly. The tagging occurs when it's added to stable tree (and potentially to apt) |
Works fine in this RPi/U-Boot pairing. |
So the set/get sequence seems to behave as expected now. Plus, it appears the behavior has reverted to what it was doing previously with 250Mhz->emmc, 100Mhz->emmc2 on rpi4b being reported as the default clock rates (without first requesting a clock rate). So Thumbs up there... That said, I've seen some odd behaviors where if I leave it sitting for a few mins in grub, then reboot it via grub's "UEFI setup", it fails to write the updated variable store to the SD card. I'm not sure if this was happening before because I wasn't using this SD card that appeared to cause the 0 returns consistently from get clock. So. I guess this fixes the primary problem, but I'm still concerned about what I've heard about SET_CLOCK, because I don't see how having the clock exceed the requested values can really be dealt with in a meaningful way. I'm generally OK with it running slower than the requested clock, but if SET_CLOCK is just a hint, and the low level firmware on the VC asks for something closer to MAX, that is going to be a problem. Particularly if it happens after we hand off control to the OS. Meaning the only "safe" way to deal with this is going to be requesting MAX clock via SET_CLOCK, which doesn't seem optimal if we are dividing it down to 50Mhz or whatever. |
Fellow @vfazio, I'm sorry, but that was my fault in patching U-Boot -- your patch is working perfectly. |
* Also revert start4.elf to most recent version as raspberrypi/firmware#1619 (#177) should be fixed now.
I think the fix has been included in a tag, at least i'm not seeing the hang as part of 1.20211029 |
In rpi-firmware 25e2b597ebfb2495eab4816a276758dcc6ea21f1, the GET_CLOCK_RATE mailbox property was changed to return the last value set by SET_CLOCK_RATE. raspberrypi/firmware#1619 (comment) Due to this change in firmware behavior, bcm2835_get_mmc_clock now returns a clock rate of zero since we do not issue SET_CLOCK_RATE. This results in degraded MMC performance. SET_CLOCK_RATE fixes the clock to a specific value and disables scaling so is not an ideal solution. Instead, fallback to GET_MAX_CLOCK_RATE in bcm2835_get_mmc_clock if GET_CLOCK_RATE returns zero. Signed-off-by: Vincent Fazio <[email protected]> Signed-off-by: Peter Robinson <[email protected]>
In rpi-firmware 25e2b597ebfb2495eab4816a276758dcc6ea21f1, the GET_CLOCK_RATE mailbox property was changed to return the last value set by SET_CLOCK_RATE. raspberrypi/firmware#1619 (comment) Due to this change in firmware behavior, bcm2835_get_mmc_clock now returns a clock rate of zero since we do not issue SET_CLOCK_RATE. This results in degraded MMC performance. SET_CLOCK_RATE fixes the clock to a specific value and disables scaling so is not an ideal solution. Instead, fallback to GET_MAX_CLOCK_RATE in bcm2835_get_mmc_clock if GET_CLOCK_RATE returns zero. Signed-off-by: Vincent Fazio <[email protected]> Signed-off-by: Peter Robinson <[email protected]>
Describe the bug
Moving Fedora from firmware 7208c3d to 25e2b59 without any other changes has regressed booting with U-Boot on Fedora.
It stops the RPi3 from booting at all and the RPi4 boots but extremely slowly. The F-34 U-Boot for RPi-3 and RPi-4
To reproduce
Expected behaviour
No "sdhci_setup_cfg: Hardware doesn't specify base clock frequency" errors and timeouts reading from the MMC
Actual behaviour
Timeouts on the mSD slot
System
Vanilla F-34 with just the RPi firmware updated.
cat /etc/rpi-issue
)? Fedora release 34 (Thirty Four)vcgencmd version
)? 2021-08-19T12:27:51 - 25e2b59uname -a
)? 5.13.12-200.fc34.aarch64Logs
Included above
Additional context
Looks most like the following but I will be checking all the in between builds:
The text was updated successfully, but these errors were encountered: