Rpi PoE hat goes brrrrr on Rpi 4b

[majkrzak]

Reposting home-assistant/operating-system#1208 as it might be base system issue

Fan of Raspberry Pi PoE HAT connected to Raspberry Pi 4 B runs constantly at full speed, no matter what options are set in config.txt. At last try it was something like: dtoverlay=rpi-poe,poe_fan_temp0=65000,poe_fan_temp0_hyst=1000,poe_fan_temp1=70000,poe_fan_temp1_hyst=2500,poe_fan_temp2=80000,poe_fan_temp2_hyst=5000,poe_fan_temp3=90000,poe_fan_temp3_hyst=5000

dmesg throws following error:

[...] rpi-poe-fan rpi-poe-fan@0: Failed to get default PWM value: -5
[...] rpi-poe-fan: probe of rpi-poe-fan@0 failed with error -5

[majkrzak]

It is failing on: https://github.com/raspberrypi/linux/blob/rpi-5.4.y/drivers/hwmon/rpi-poe-fan.c#L329

[majkrzak]

It can be replicated with latest raspbian: https://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2021-01-12/2021-01-11-raspios-buster-armhf-full.zip

[majkrzak]

Updating /boot/ with this from /dev/ branch does not help

[shantanugoel]

I ran into the same issue. I use archlinux which packages the raspberrypi-bootloader and raspberrypi-bootloader-x packages. Downgrading back to the 20210111 version fixed this for me.

[ndanyluk]

Echoing that I am seeing this on 3x Pi 3B+ and 1x Pi 4B all simultaneously after upgrading kernel to 5.10.y using apt update && apt upgrade -y. However, in my case I get no fan control at all and the Pis are thermally throttling.

No changes made to config.txt from stock, stock Raspbian lite, same errors in dmesg

[shantanugoel]

My hunch is that this issue is related to commit c78f3ef which has some changes for poe_hat/i2c

[pelwell]

I've independently arrived at the same conclusion, and reverting that commit make the POE hat fan work again, but with other consequences. The author of that commit is aware.

[pelwell]

A fix is in the works, but in the meantime you can revert the firmware to before the problem was introduced:

$ sudo SKIP_KERNEL=1 rpi-update 32f92809

[majkrzak]

I'm wondering how I encounter this issue with Home Assistant Rpi4 image 5.10 released on 1st of January while problematic commit is from 15th? 🤔

[EdBoraas]

I'm wondering how I encounter this issue with Home Assistant Rpi4 image 5.10 released on 1st of January while problematic commit is from 15th? 🤔

@majkrzak, it's possible that your original issue was actually something different. One thing that bit me in the past is that the dtoverlay= lines in config.txt have a line length limit, and they just ignore anything past that limit. I had a very similar issue to you, in that poe_fan_temp3 wasn't being set because it was past the line length, so I was nearly-continually in excess of the default amount for temp3, causing my fan to run excessively.

You can verify that your temperature and hysteresis values are set correctly by running this command:
od -An --endian=big -td4 /proc/device-tree/thermal-zones/cpu-thermal/trips/trip?/temperature /proc/device-tree/thermal-zones/cpu-thermal/trips/trip?/hysteresis
(od is necessary in this case to decode the raw values as presented via the /proc interface)

The solution was to break out the dtparam= values onto separate lines, like so:

dtoverlay=rpi-poe   
dtparam=poe_fan_temp0=65000   
dtparam=poe_fan_temp0_hyst=1000   
dtparam=poe_fan_temp1=70000
dtparam=poe_fan_temp1_hyst=2500
dtparam=poe_fan_temp2=80000
dtparam=poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=90000
dtparam=poe_fan_temp3_hyst=5000

Hopefully that helps. I suspect the two issues may have gotten conflated here; in my experience, the recently-introduced firmware issue seems to force the fan off. not on.

[jordipalet]

A fix is in the works, but in the meantime you can revert the firmware to before the problem was introduced:

$ sudo SKIP_KERNEL=1 rpi-update 32f92809

Trying to use this, however, it turns into a download of 35 minutes (for about 114M) then I don't think it is completed correctly because I get:
gzip: stdin: unexpected end of file
tar: unexpected end of file
tar: Error is not recoverable: exiting now

Any suggestion ?

[pelwell]

That sounds like one of the following:

• Flaky internet connection.
• Aging SD card.
• Full SD card.