Raspberry Pi PoE hat fans don't respect /boot/config.txt settings

[neilgoodman]

Is this the right place for my bug report?
This repository contains the GPU firmware used on the Raspberry Pi. This software is the closed source part of the Raspberry Pi system, it includes booting (including network booting and USB booting), low-level power and clock control, FKMS and legacy HDMI control (not full KMS that is in the linux kernel), hardware legacy codecs (MPEG2, H264 and VC1), encode hardware including the ISP (image sensor pipeline) and camera control, audio output (analogue and HDMI audio).

If you believe that the issue you are seeing is within this area, this is the right place. If not, we have other repositories for the linux kernel at github.com/raspberrypi/linux and Raspberry Pi userland applications at github.com/raspberrypi/userland. If you have problems with the Raspbian distribution packages, report them in the github.com/RPi-Distro/repo. If you simply have a question, then the Raspberry Pi forums are the best place to ask it.

Describe the bug
I'm using the official Raspberry Pi PoE Hat (the first gen, not the plus version) with a Raspberry Pi 4 Model B Rev 1.4.

I recently applied an update from my distro and when I rebooted the machine I noticed the fan on the PoE hat started running and didn't turn off. I have added this block to my /boot/config.txt to prevent the fan from running in idle conditions:

dtparam=poe_fan_temp0=65000,poe_fan_temp0_hyst=1000
dtparam=poe_fan_temp1=70000,poe_fan_temp1_hyst=5000
dtparam=poe_fan_temp2=75000,poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=80000,poe_fan_temp3_hyst=2000

I checked by the CPU temperature and got this output:

$ vcgencmd measure_temp
temp=55.5'C

The firmware I upgraded to was this version:

Jan 24 2022 18:00:30
Copyright (c) 2012 Broadcom
version 94562b1518ca82ece28042cca1e5cdbbb43c8bda (clean) (release) (start)

So, I downgraded my firmware to this version:

Jan 20 2022 13:56:48
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

And after reboot, the PoE hat turned off as expected.

To reproduce
Note: It's assumed you are indoors in a room at room temperature.

1. Install an official PoE hat onto a Raspberry Pi 4 Model B Rev 1.4
2. Install firmware version 94562b1518ca82ece28042cca1e5cdbbb43c8bda
3. Add the following to /boot/config.txt and reboot

dtparam=poe_fan_temp0=65000,poe_fan_temp0_hyst=1000
dtparam=poe_fan_temp1=70000,poe_fan_temp1_hyst=5000
dtparam=poe_fan_temp2=75000,poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=80000,poe_fan_temp3_hyst=2000

Expected behaviour
After reboot, the PoE hat fan should turn off

Actual behaviour
After reboot, the PoE hat fan stays on

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW

  • Raspberry Pi 4 Model B Rev 1.4

• Which OS and version (cat /etc/rpi-issue)?

  • ArchLinux Arm using the following packages:
    • linux-rpi
    • raspberrypi-bootloader

• Which firmware version (vcgencmd version)?

$ vcgencmd version
Copyright (c) 2012 Broadcom
version 94562b1518ca82ece28042cca1e5cdbbb43c8bda (clean) (release) (start)

• Which kernel version (uname -a)?

$ uname -a
Linux 5.10.94-1-rpi-ARCH #1 SMP PREEMPT Thu Jan 27 20:16:30 UTC 2022 aarch64 GNU/Linux

Logs
If applicable, add the relevant output from dmesg or similar.

Additional context
Add any other relevant context for the problem.

[pelwell]

Please post your full config.txt and the output of sudo vcdbg log msg.

[neilgoodman]

config.txt:

# See /boot/overlays/README for all available options

dtoverlay=vc4-kms-v3d
initramfs initramfs-linux.img followkernel

arm_64bit=1
disable_splash=1

# disable bluetooth
dtoverlay=disable-bt

# disable wifi
dtoverlay=disable-wifi

# fan settings
# see https://github.com/raspberrypi/firmware/blob/master/boot/overlays/README#L2480-L2498
dtparam=poe_fan_temp0=65000,poe_fan_temp0_hyst=1000
dtparam=poe_fan_temp1=70000,poe_fan_temp1_hyst=5000
dtparam=poe_fan_temp2=75000,poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=80000,poe_fan_temp3_hyst=2000

[pi4]
# Run as fast as firmware / board allows
arm_boost=1

sudo vcdbg log msg:

I don't have access to vcdbg, I think because I'm on a 64-bit kernel and I'm blocked by this issue: #1118

There is a workaround posted in that issue, but it looks like it's for Debian/Ubuntu and it'll take me a little bit to see if I can get it working on Arch.

[pelwell]

Although the overlays for the PoE hats are loaded automatically, if you want to configure them by supplying parameters it is necessary to put the parameters "in scope" by loading the overlay explicitly immediately before using them, i.e. add one of the following just before the reference to poe_fan_temp0:

dtoverlay=rpi-poe
dtoverlay=rpi-poe-plus

[neilgoodman]

@pelwell I tried adding dtoverlay=rpi-poe before the dtparam=poe_fan_temp0 line, but it didn't help. Also, on previous versions of the firmware, dtoverlay=rpi-poe was not required.

I'm going to install Raspberry Pi OS today and see if I can repro the issue. If so, I'll use the commands recommended above to provide more logs.

[neilgoodman]

I'm having a different issue on 64-bit Raspberry Pi OS, the fans don't turn on even though my temp reading is this:

$ vcgencmd measure_temp
temp=80.3'C

If I use 32-bit Raspberry Pi OS, it works as expected.

OS:

Raspberry Pi reference 2022-01-28
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, fbe448ccdc995d295d24c7596e5f0ef62cc2488f, stage4

Kernel:

Linux raspberrypi 5.10.94-v8+ #1518 SMP PREEMPT Thu Jan 27 14:55:19 GMT 2022 aarch64 GNU/Linux

Firmware:

Copyright (c) 2012 Broadcom
version 94562b1518ca82ece28042cca1e5cdbbb43c8bda (clean) (release) (start)

Reproduction Steps:

Rapsberry Pi OS Image: https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64.zip

After installing the OS, I booted and signed in via SSH, and then I cloned this repo and checked out commit b76ea7f513219aa2c0a28dfcd30c6962152cca9d. I then copied firmware/boot to /boot and I updated config.txt to the following and then rebooted:

# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# Enable DRM VC4 V3D driver
dtoverlay=vc4-kms-v3d
max_framebuffers=2

# Run in 64-bit mode
arm_64bit=1

# Disable compensation for displays with overscan
disable_overscan=1

# fan settings
# see https://github.com/raspberrypi/firmware/blob/master/boot/overlays/README#L2480-L2498
dtoverlay=rpi-poe
dtparam=poe_fan_temp0=65000,poe_fan_temp0_hyst=1000
dtparam=poe_fan_temp1=70000,poe_fan_temp1_hyst=5000
dtparam=poe_fan_temp2=75000,poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=80000,poe_fan_temp3_hyst=2000

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[all]

[pi4]
# Run as fast as firmware / board allows
arm_boost=1

[all]

[neilgoodman]

I followed the workaround in #1118 and got vcdbg working on 64-bit Raspberry Pi OS. I also reinstalled everything and I am now using commit 48606ad.

I'm still reproducing the same issue above. The rpi-poe fans do not turn on, even if I set the config.txt values to turn them on at the highest level with a low value (I'm setting the temp to 20000).

@pelwell do the debug logs below help?

Details:

CPU Temp:

$ vcgencmd measure_temp
temp=50.1'C

OS:

Raspberry Pi reference 2022-01-28
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, fbe448ccdc995d295d24c7596e5f0ef62cc2488f, stage2

Kernel:

Linux raspberrypi 5.15.18-v8+ #1520 SMP PREEMPT Fri Feb 4 12:04:45 GMT 2022 aarch64 GNU/Linux

Firmware:

Feb  4 2022 12:05:40 
Copyright (c) 2012 Broadcom
version a26faf97e3bf76bcc23949d7cdab2f96f399a0c3 (clean) (release) (start)

sudo vcdbg log msg output:

https://pastebin.com/qkw4i223

/boot/config.txt:

# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# Enable DRM VC4 V3D driver
dtoverlay=vc4-kms-v3d
max_framebuffers=2

# Run in 64-bit mode
arm_64bit=1

# Disable compensation for displays with overscan
disable_overscan=1

dtoverlay=rpi-poe
dtparam=poe_fan_temp0=20000,poe_fan_temp0_hyst=1000
dtparam=poe_fan_temp1=20000,poe_fan_temp1_hyst=5000
dtparam=poe_fan_temp2=20000,poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=20000,poe_fan_temp3_hyst=2000

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[all]

[pi4]
# Run as fast as firmware / board allows
arm_boost=1

[all]

[pelwell]

Thanks - I see what is happening. The point where the automatic loading of the rpi-poe overlay happens is after the point where config.txt is scanned for overlays, causing your settings to be overwritten. I'm sure this was not the case previously - I've tested the fan settings before. Leave this with me.

[6by9]

Catch 22 - it needs to be late enough to detect that an overlay has enabled i2c_vc, but that also means that it's after loading the overrides.
If loading the overlay manually then it likely also needs the i2c override.

[pelwell]

It may be possible to rework it in such a way that the switch can be made after the overlay has been merged.

[pelwell]

Linux pull request raspberrypi/linux#4877 is half of the proposed fix, the other half being a firmware patch.

[laha83]

I am running a pi4 8gb 32bit piOS and the poe fan do not respect the config.txt overrides

Copyright (c) 2012 Broadcom
version a26faf97e3bf76bcc23949d7cdab2f96f399a0c3 (clean) (release) (start)

Linux WEGA-iobroker 5.15.24-v7l+ #1526 SMP Fri Feb 18 14:57:32 GMT 2022 armv7l GNU/Linux
and parameters are still overwritten by defaults.

here is my config.txt:

# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
#disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800
arm_boost=1


# Uncomment some or all of these to enable the optional hardware interfaces
dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
#dtparam=audio=on

#dtoverlay=vc4-kms-v3d
#dtoverlay=vc4-fkms-v3d
dtoverlay=disable-wifi
dtoverlay=disable-bt


# PoE Hat Fan Speeds
dtoverlay=rpi-poe
dtparam=poe_fan_temp0=10000,poe_fan_temp0_hyst=1000
dtparam=poe_fan_temp1=55000,poe_fan_temp1_hyst=5000
dtparam=poe_fan_temp2=65000,poe_fan_temp2_hyst=5000
dtparam=poe_fan_temp3=68000,poe_fan_temp3_hyst=5000
dtparam=poe_fan_temp4=70000,poe_fan_temp4_hyst=5000

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
#dtoverlay=vc4-fkms-v3d
#max_framebuffers=2

Here is the output of od:

~ $ od -An --endian=big -td4 /proc/device-tree/thermal-zones/cpu-thermal/trips/trip?/temperature /proc/device-tree/thermal-zones/cpu-thermal/trips/trip?/hysteresis
       40000       45000       50000       55000
        2000        2000        2000        5000

So I think the issue is still alive.
Do you need more information?

[pelwell]

The problem hasn't been fixed because although the companion patch went into our internal firmware software repo on February 9th, the last firmware build (the one you are running) was released on February 4th - the last kernel build was much more recent (the 18th).

Any chance of a firmware release, @popcornmix ?

[brettonw]

Is this fix workable for Ubuntu 21, or did the mechanism for controlling the fan speed change?

[6by9]

Is this fix workable for Ubuntu 21, or did the mechanism for controlling the fan speed change?

That's up to whether Ubuntu have picked up the relevant kernel and firmware updates - Raspberry Pi have no control over that.