Rpi PoE hat goes brrrrr on Rpi 4b

[majkrzak]

Reposting home-assistant/operating-system#1208 as it might be base system issue

Fan of Raspberry Pi PoE HAT connected to Raspberry Pi 4 B runs constantly at full speed, no matter what options are set in config.txt. At last try it was something like: dtoverlay=rpi-poe,poe_fan_temp0=65000,poe_fan_temp0_hyst=1000,poe_fan_temp1=70000,poe_fan_temp1_hyst=2500,poe_fan_temp2=80000,poe_fan_temp2_hyst=5000,poe_fan_temp3=90000,poe_fan_temp3_hyst=5000

dmesg throws following error:

[...] rpi-poe-fan rpi-poe-fan@0: Failed to get default PWM value: -5
[...] rpi-poe-fan: probe of rpi-poe-fan@0 failed with error -5

[majkrzak]

It is failing on: https://github.com/raspberrypi/linux/blob/rpi-5.4.y/drivers/hwmon/rpi-poe-fan.c#L329

[majkrzak]

It can be replicated with latest raspbian: https://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2021-01-12/2021-01-11-raspios-buster-armhf-full.zip

[majkrzak]

Updating /boot/ with this from /dev/ branch does not help

[shantanugoel]

I ran into the same issue. I use archlinux which packages the raspberrypi-bootloader and raspberrypi-bootloader-x packages. Downgrading back to the 20210111 version fixed this for me.

[ndanyluk]

Echoing that I am seeing this on 3x Pi 3B+ and 1x Pi 4B all simultaneously after upgrading kernel to 5.10.y using apt update && apt upgrade -y. However, in my case I get no fan control at all and the Pis are thermally throttling.

No changes made to config.txt from stock, stock Raspbian lite, same errors in dmesg

[shantanugoel]

My hunch is that this issue is related to commit c78f3ef which has some changes for poe_hat/i2c

[pelwell]

I've independently arrived at the same conclusion, and reverting that commit make the POE hat fan work again, but with other consequences. The author of that commit is aware.

[pelwell]

A fix is in the works, but in the meantime you can revert the firmware to before the problem was introduced:

$ sudo SKIP_KERNEL=1 rpi-update 32f92809

[majkrzak]

I'm wondering how I encounter this issue with Home Assistant Rpi4 image 5.10 released on 1st of January while problematic commit is from 15th? 🤔

[EdBoraas]

I'm wondering how I encounter this issue with Home Assistant Rpi4 image 5.10 released on 1st of January while problematic commit is from 15th? 🤔

@majkrzak, it's possible that your original issue was actually something different. One thing that bit me in the past is that the dtoverlay= lines in config.txt have a line length limit, and they just ignore anything past that limit. I had a very similar issue to you, in that poe_fan_temp3 wasn't being set because it was past the line length, so I was nearly-continually in excess of the default amount for temp3, causing my fan to run excessively.

You can verify that your temperature and hysteresis values are set correctly by running this command:
od -An --endian=big -td4 /proc/device-tree/thermal-zones/cpu-thermal/trips/trip?/temperature /proc/device-tree/thermal-zones/cpu-thermal/trips/trip?/hysteresis
(od is necessary in this case to decode the raw values as presented via the /proc interface)

The solution was to break out the dtparam= values onto separate lines, like so:

dtoverlay=rpi-poe   
dtparam=poe_fan_temp0=65000   
dtparam=poe_fan_temp0_hyst=1000   
dtparam=poe_fan_temp1=70000
dtparam=poe_fan_temp1_hyst=2500
dtparam=poe_fan_temp2=80000
dtparam=poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=90000
dtparam=poe_fan_temp3_hyst=5000

Hopefully that helps. I suspect the two issues may have gotten conflated here; in my experience, the recently-introduced firmware issue seems to force the fan off. not on.

[jordipalet]

A fix is in the works, but in the meantime you can revert the firmware to before the problem was introduced:

$ sudo SKIP_KERNEL=1 rpi-update 32f92809

Trying to use this, however, it turns into a download of 35 minutes (for about 114M) then I don't think it is completed correctly because I get:
gzip: stdin: unexpected end of file
tar: unexpected end of file
tar: Error is not recoverable: exiting now

Any suggestion ?

[pelwell]

That sounds like one of the following:

• Flaky internet connection.
• Aging SD card.
• Full SD card.

[pelwell]

For those of you reporting no fan activity, this has been diagnosed and a fix pushed to the internal firmware repo. Expect a release in a few days.

[jordipalet]

Tks! after 2 a couple of hours and a few retries, it worked ...

Internet was 300 gbps symmetrical, not using an SD, but an external USB3.1 SSD with free space ...

So I guess it was something on the other side (github servers) or just bad luck.

Normally any firmware update takes a couple of seconds, so don't understand why this "downgrade" took more than half an hour ....

[jordipalet]

For those of you reporting no fan activity, this has been diagnosed and a fix pushed to the internal firmware repo. Expect a release in a few days.

I will try next saturday ... should be done by then! avoid risks during the week ...

[popcornmix]

rpi-update contains the potential fix. Please test and report.

[majkrzak]

As I had feelings that problem with my device had different reason (maybe hardware) I decided to get replacement, so I'm not anymore able to check the fix.

[FrankMaute]

rpi-update contains the potential fix. Please test and report.

That worked, thanks.

[Mark-Spitz]

rpi-update contains the potential fix. Please test and report.

My two RPI are working fine again (5.10.14). From temp=70 back to 45.. Pffffff Thanks a lot!

[cubecell]

The fix works for my RPI 3B+, but the PoE-Fan on my new RPI 4B still runs at full speed all the time.
On the RPI 3 I found a cooling_device0 and in /sys/class/thermal, on the RPI 4 there is only the thermal_zone0

[EdBoraas]

rpi-update contains the potential fix. Please test and report.

@popcornmix Working well with the fix via rpi-update. Thanks.

[jordipalet]

rpi-update contains the potential fix. Please test and report.

@popcornmix Working well with the fix via rpi-update. Thanks.

I'm confused that some people said it is working, others not. I think it will be good to be more specific, such as:
a) It is working in Rpi4b?
b) It means the fan speed is still controllable as it was previously or will keep running all the time?

[pelwell]

I specifically tested it on Pi 4 - can you post your non-standard config.txt settings?

[cubecell]

I only add

dtoverlay=rpi-poe  
dtparam=poe_fan_temp0=45000
dtparam=poe_fan_temp1=50000  
dtparam=poe_fan_temp2=55000  
dtparam=poe_fan_temp3=60000

dmesg shows the same error:

[...] rpi-poe-fan rpi-poe-fan@0: Failed to get default PWM value: -5
[...] rpi-poe-fan: probe of rpi-poe-fan@0 failed with error -5

[pelwell]

That suggests you are still running the old firmware. What does vcgencmd version report?

[ricktendo]

Works now

[jordipalet]

I know. I didn't done both of them at the same time ... just in different reboots, just to try different things. After deleting the .firmware_revision, it seems is working correctly (even after a new apt upgrade ... Anyway, all fine now, at least at the time being.
Tks!

[tdolder]

rpi-update

Removing .firmware_revision and performing the firmware update again did not solve the issue.

[ 7.660334] rpi-poe-fan rpi-poe-fan@0: Failed to get default PWM value: -5
[ 7.660387] rpi-poe-fan: probe of rpi-poe-fan@0 failed with error -5

[pelwell]

When did it last work? You still haven't identified the point at which it went wrong.

[jordipalet]

yes, deleting .firmware_revision and running again rpi-update, worked! Only different thing vs previous attempts was deleting .firmware_revision

[pelwell]

That's great - I was specifically addressing @tdolder.

[tdolder]

That's great - I was specifically addressing @tdolder.

It never worked. It is a new Raspi 3B + with official Raspi POE hat running Octoprint.
First tried the Raspi-config option for the fan control. This didn't work. Then tried via dtoverlay in the config.txt.
I am now going to try a clean install.

[pelwell]

If you have a spare card, install a fresh Raspberry Pi OS, run sudo rpi-update and add just dtoverlay=rpi-poe to config.txt. If that doesn't work then you have a hardware problem.

[tdolder]

If you have a spare card, install a fresh Raspberry Pi OS, run sudo rpi-update and add just dtoverlay=rpi-poe to config.txt. If that doesn't work then you have a hardware problem.

Test done > spare card with PiOS lite:
After sudo rpi-update and reboot, no "rpi-poe-fan" errors in dmesg
After adding the 'dtoverlay=rpi-poe' to the config.txt the errors appear,

[ 6.879401] rpi-poe-fan rpi-poe-fan@0: Failed to get default PWM value: -5
[ 6.879446] rpi-poe-fan: probe of rpi-poe-fan@0 failed with error -5

Hardware issue with PoE HAT or Raspi?

[cubecell]

In my case it was a hardware failure of the PoE HAT.
New HAT --> no dmesg errors anymore !
@pelwell many thanks for everything !

[pelwell]

I wouldn't necessarily get to hear about failures in the field, but in my opinion it is more likely to be the HAT just because there is more to go wrong. The header pins are wired directly to the pins/balls on the SoC, so unless there's a broken/shorted track or the Pi has been subjected to high voltages (5V is fatal to most pins), there isn't much that can fail.

[tdolder]

@pelwell thanks for your help. I am going to request a replacement for my PoE hat.

[pelwell]

I think that's reasonable, under the circumstances - the problem has been fixed for everyone else on this issue, and @cubecell has proved that PoE HAT faults are possible and have the same symptoms.

[MichalMMac]

PoE HAT fan started working again after manually upgrading the firmware to the latest beta using rpi-update. Thanks @popcornmix :-)

[simcdev]

I can confirm that on a fresh Raspberry Pi OS Lite (32-bit), Released 2011-01-11 with a apt update and apt full-upgrade the PoE HAT fan does not spin.
After rpi-update and reboot, the fan on PoE HAT starts again. No edit in /boot/config.txt

[pelwell]

The firmware fixes will be included in future apt package releases. Apart from a report of a problem with the 64-bit OS, I think we're done here.

[tdolder]

@pelwell thanks for your help. I am going to request a replacement for my PoE hat.

Yesterday I received a new PoE-Hat from my supplier. No more error messages. So the problem was a defective PoE-Hat.

[yannickgranger]

RPi 4B 8Gb - fresh Rasperry Pi Os 32 bit 5.15.50-v7l+
Ran rpi-update and added dtoverlay=rpi-poe to config.txt
Did not work, so changed the Hat for a new one, still in error :

dmesg output :

[ 6.037744] raspberrypi-poe-pwm soc:firmware:pwm: Failed to get duty cycle: -EIO
[ 6.037767] raspberrypi-poe-pwm: probe of soc:firmware:pwm failed with error -5

[Celos]

Rpi 3B+ with the original poe hat, latest Octopi stable image (0.18.0) with everything updated.

Started out with:

[ 7.098774] rpi-poe-fan rpi-poe-fan@0: Failed to get default PWM value: -5
[ 7.098832] rpi-poe-fan: probe of rpi-poe-fan@0 failed with error -5

After running rpi-update:

[ 7.983454] raspberrypi-poe-pwm soc:firmware:pwm: Failed to get duty cycle: -EIO
[ 7.983555] raspberrypi-poe-pwm: probe of soc:firmware:pwm failed with error -5

[ianhundere]

same as @yannickgranger and @Celos here as well.

[    5.830427] raspberrypi-poe-pwm soc:firmware:pwm: Failed to get duty cycle: -EIO
[    5.830463] raspberrypi-poe-pwm: probe of soc:firmware:pwm failed with error -5

edit: after further investigation i realized my ground wires were not attached correctly. all is well here.

[pelwell]

An original PoE hat is working fine here on a 3B+, both with the current 5.15 kernel and with the 6.1 kernel from sudo rpi-update next. There are no relevant error or warnings messages.

Note that it should not be necessary to explicitly load the dtoverlay - the firmware should do that automatically - but doing so should not be harmful.

Has the fan on the PoE HAT ever worked for you?

[ianhundere]

@pelwell that's the thing, i've had these (a total of 4) for a couple of years and they've never turned on before. i just never got around to troubleshooting it. it's hard to think that 4 went bad, so maybe they're not connected correctly or ?

i'm using mine with PI 4 MODEL B, 4GB.

this is the poe hat:

• https://www.amazon.com/gp/product/B07WD7HXSQ

edit: i'm gonna investigate further and see if i'm connecting the fan incorrectly.

[Celos]

For me the issue is that the fan never turns off. It's stays on even when I shut off the rpi.

[pelwell]

@ianhundere That isn't one of our PoE HATs, so it's a stretch to say "same as" the other two users. Does that board even provide PWM fan control, rather than just a simple power signal? The lack of a PWM fan controller would almost certainly lead to the error messages you are seeing.

[pelwell]

This RPi PoE HAT does stop when the Pi shuts down, on both a 3B+ and a 4B. There are no errors or warning in the log.

Since you've run an rpi-update recently you will be running the latest firmware and kernel, which tends to point towards it being a hardware problem.

Full disclosure - my setup is still being powered by the usual USB power socket rather than PoE, but a PoE injector is on order.

[ianhundere]

@ianhundere Does that board even provide PWM fan control, rather than just a simple power signal?

i believe it does, at least that's what its documentation is pointing towards. thanks for the info.

[pelwell]

I did look for documentation but failed to find any.

[ianhundere]

for documentation but failed to find any.

heh, i just looked at amazon pics from customers, and after further investigation, i didn't have my ground wires connected properly. so it's all good here 🎉.

[pelwell]

@Celos In tests with the new PoE injector the fan shuts off with the Pi. Which kernel are you running?

[Celos]

pi@octopi:~ $ uname -a
Linux octopi 5.15.80-v7+ #1602 SMP Tue Nov 29 14:42:58 GMT 2022 armv7l GNU/Linux

I just bit the bullet and bought a new PoE+ version, though, as prices have finally come down. It seems to work fine: the fan starts and stops as expected (Absolutely insane coil whine when I shut it down, though). I guess the hardware was just faulty.

[ianhundere]

just wanted to chime back in and say that my fan was not responding to temp settings, but after running rpi-update i'm in business.

edit: furthermore, i also do not have a poe hat w/ fan control which explains why my temp settings have not been working. will upgrade and report back.