Cannot disable EEE on Raspberry Pi 5

[b4dpxl]

Describe the bug

My Raspberry Pi5, running bookworm, is experiencing frequent eth0 droppages. Example:

[Mar23 13:30] macb 1f00100000.ethernet eth0: Link is Down
[ +38.912436] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 00:23] macb 1f00100000.ethernet eth0: Link is Down
[  +3.072515] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 02:17] macb 1f00100000.ethernet eth0: Link is Down
[  +3.072523] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 14:31] macb 1f00100000.ethernet eth0: Link is Down
[Mar24 14:32] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 16:54] macb 1f00100000.ethernet eth0: Link is Down
[  +3.068473] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 18:58] macb 1f00100000.ethernet eth0: Link is Down
[  +3.072540] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

I've changed switch ports, cables, etc. This was also an issue on RPi 4, which could be resolved by disabling EEE, either with ethtool or dtparam. I want to see if this also solves the problem on Pi5, but neither of these options work on RPi5, and trying to run ethtool reports "Operation not supported".

See https://forums.raspberrypi.com/viewtopic.php?t=360924

Can EEE be disabled on RPi5 via another method, or is there an ETA when disabling it will be supported on Pi5?

Steps to reproduce the behaviour

run ethtool --set-eee eth0 eee off.

Get the response: netlink error: Operation not supported

Device (s)

Raspberry Pi 5

System

# cat /etc/rpi-issue
Raspberry Pi reference 2023-12-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 2acf7afcba7d11500313a7b93bb55a2aae20b2d6, stage2

# vcgencmd version
2023/11/20 19:40:17 
Copyright (c) 2012 Broadcom
version 361205c6 (release) (embedded)

# uname -a
Linux rocinante 6.1.0-rpi7-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux

Logs

No response

Additional context

No response

[alessandromrc]

If your problem is the link always negotiating, you can just use these two commands here:

sudo ethtool -s eth0 autoneg off
sudo ethtool -s eth0 speed 1000

Obviously set the speed as the one you want, for 1Gbps you will put 1000, like I've already done in the example.

Also ensure to run the command as a root user / using sudo since you cannot do these changes with a normal user.

[b4dpxl]

I don't believe it's a negotiation issue; I'd already checked playing with ethtool settings before hand. It still happened with 1gb autoneg off, but was fine when forced to 100mb with/without autoneg. This mirrors what other people had previously observed on Pi4's.

I was able to borrow a different switch (Netgear, same as my current one, but slightly newer version) and so far it's been fine.

This also mirrors what people observed on Pi4, where it seems to be an incompatibility with EEE between the Pi and some switch models.

Again, I can't confirm this as it's impossible to turn off EEE (AFAIK) on the Pi 5

See here as examples of the issue and here on Pi4

[alessandromrc]

Interesting to know @b4dpxl, that seems to be related more on the switch side, I know not being able to disable EEE can be seen as a bad thing but I also see no reason of why it would hurt not having that... the problem comes with non-compliant network equipment...

If you got the model of such switch let us know, would be nice to have a list around so people having connection issues know the reason...

Also EEE is mainly hardware related and I am not really sure if the chip used in Pi4/5 even allow such feature.

PS. Everything is fine here with a Cisco Catalyst switch, in 100 and 1000 modes.

[b4dpxl]

In my case the problematic switch is a Netgear GS108Ev3, but a newer version Netgear GS108PEv3 is fine. I suspect most Pi users aren't running Cisco network equipment though 🤨

Well, as I understand it, the Pi4 and Pi5 both share the ethernet chip, and the Pi4 definitely allows you to disable EEE. It's referenced here in the overlays document and here is the pull request which was merged to allow it to be disabled via dtparam.

Clearly this was a known issue, otherwise this wouldn't have been added. As per the pull:

The EEE implementation on BCM2711 doesn't work well with all switches.

IMO it's not fair to place the blame on the switch when there are multiple reported switches at issue, and one common factor (the Pi). Did you actually look at any of the links I provided before?

[alessandromrc]

In my case the problematic switch is a Netgear GS108Ev3, but a newer version Netgear GS108PEv3 is fine. I suspect most Pi users aren't running Cisco network equipment though 🤨

Well, as I understand it, the Pi4 and Pi5 both share the ethernet chip, and the Pi4 definitely allows you to disable EEE. It's referenced here in the overlays document and here is the pull request which was merged to allow it to be disabled via dtparam.

Clearly this was a known issue, otherwise this wouldn't have been added. As per the pull:

The EEE implementation on BCM2711 doesn't work well with all switches.

IMO it's not fair to place the blame on the switch when there are multiple reported switches at issue, and one common factor (the Pi). Did you actually look at any of the links I provided before?

Yes, I've looked into the links you shared but not really deeply since I had not much time, anyways since you mentioned that it can be disabled on Pi4, I will later test out on a 4 and see the results I get and if there's something possible in the 5...

[alessandromrc]

Just tested on a 4 and I can see the NIC shutting down and powering back on... Will check later what could cause the problem on the 5.

[alessandromrc]

Turns out I've found difference between the Pi4 and the Pi5, if you run "lspci" you can see that on the Pi5 the ethernet controller is connected through PCI unlike the 4...

I think this is causing some issues to ethtool at this moment, will continue to look around.

PS. I think that the PCI ethernet controller is practically the new RP1 chip.

[alessandromrc]

Also @b4dpxl if you're interested into the RP1 chip you can find some infos here: https://datasheets.raspberrypi.com/rp1/rp1-peripherals.pdf

Sadly there's nothing about EEE so I guess we can do nothing at this moment : /

[alessandromrc]

After looking around the current implementation of Ethernet on RP1 doesn't support EEE at all since the implementation of it is included in IEEE 802.3az, meanwhile on the RP1 docs it is just IEEE 802.3...

Not sure if this will be changed in the future but I doubt anything will come soon, even because EEE is a good feature to always have enabled.

[pelwell]

I have seen no mention of EEE or 802.3az in the documentation for the Ethernet MAC in RP1.

[SensorAnalyticsAus]

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:

scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed

Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.

Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

[alessandromrc]

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:

scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed

Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.

Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

[SensorAnalyticsAus]

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   12GB 105.8MB/s   02:00


scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                        19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe

*** Connection stalled in just over 2 mins of data transfer

MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  41.5MB/s   22:00

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  42.8MB/s   21:20

*** Connection stable over 43 mins of data transfer

In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?

[alessandromrc]

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   12GB 105.8MB/s   02:00


scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                        19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe

*** Connection stalled in just over 2 mins of data transfer

MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  41.5MB/s   22:00

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  42.8MB/s   21:20

*** Connection stable over 43 mins of data transfer

In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?

I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...

Either ways, could you run the command:

uname -a

So we can at least understand what kernel you are running on, the output in my case is this one:
Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux

Just finished transferring 100GBs... I guess the problem is not happening here at all:

alessandro@alessandro-laptop:~$ scp alessandro@raspberrypi5.local:/home/alessandro/test ./
alessandro@raspberrypi5.local's password: 
test                                                                                                                                                100%  100GB  93.6MB/s   18:14