Skip to content

Cannot disable EEE on Raspberry Pi 5 #6065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
b4dpxl opened this issue Mar 24, 2024 · 26 comments
Closed

Cannot disable EEE on Raspberry Pi 5 #6065

b4dpxl opened this issue Mar 24, 2024 · 26 comments

Comments

@b4dpxl
Copy link

b4dpxl commented Mar 24, 2024

Describe the bug

My Raspberry Pi5, running bookworm, is experiencing frequent eth0 droppages. Example:

[Mar23 13:30] macb 1f00100000.ethernet eth0: Link is Down
[ +38.912436] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 00:23] macb 1f00100000.ethernet eth0: Link is Down
[  +3.072515] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 02:17] macb 1f00100000.ethernet eth0: Link is Down
[  +3.072523] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 14:31] macb 1f00100000.ethernet eth0: Link is Down
[Mar24 14:32] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 16:54] macb 1f00100000.ethernet eth0: Link is Down
[  +3.068473] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 18:58] macb 1f00100000.ethernet eth0: Link is Down
[  +3.072540] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

I've changed switch ports, cables, etc. This was also an issue on RPi 4, which could be resolved by disabling EEE, either with ethtool or dtparam. I want to see if this also solves the problem on Pi5, but neither of these options work on RPi5, and trying to run ethtool reports "Operation not supported".

See https://forums.raspberrypi.com/viewtopic.php?t=360924

Can EEE be disabled on RPi5 via another method, or is there an ETA when disabling it will be supported on Pi5?

Steps to reproduce the behaviour

run ethtool --set-eee eth0 eee off.

Get the response: netlink error: Operation not supported

Device (s)

Raspberry Pi 5

System

# cat /etc/rpi-issue
Raspberry Pi reference 2023-12-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 2acf7afcba7d11500313a7b93bb55a2aae20b2d6, stage2

# vcgencmd version
2023/11/20 19:40:17 
Copyright (c) 2012 Broadcom
version 361205c6 (release) (embedded)

# uname -a
Linux rocinante 6.1.0-rpi7-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux

Logs

No response

Additional context

No response

@alessandromrc
Copy link

alessandromrc commented Mar 28, 2024

If your problem is the link always negotiating, you can just use these two commands here:

sudo ethtool -s eth0 autoneg off
sudo ethtool -s eth0 speed 1000

Obviously set the speed as the one you want, for 1Gbps you will put 1000, like I've already done in the example.

Also ensure to run the command as a root user / using sudo since you cannot do these changes with a normal user.

@b4dpxl
Copy link
Author

b4dpxl commented Mar 28, 2024

I don't believe it's a negotiation issue; I'd already checked playing with ethtool settings before hand. It still happened with 1gb autoneg off, but was fine when forced to 100mb with/without autoneg. This mirrors what other people had previously observed on Pi4's.

I was able to borrow a different switch (Netgear, same as my current one, but slightly newer version) and so far it's been fine.

This also mirrors what people observed on Pi4, where it seems to be an incompatibility with EEE between the Pi and some switch models.

Again, I can't confirm this as it's impossible to turn off EEE (AFAIK) on the Pi 5

See here as examples of the issue and here on Pi4

@alessandromrc
Copy link

alessandromrc commented Mar 28, 2024

Interesting to know @b4dpxl, that seems to be related more on the switch side, I know not being able to disable EEE can be seen as a bad thing but I also see no reason of why it would hurt not having that... the problem comes with non-compliant network equipment...

If you got the model of such switch let us know, would be nice to have a list around so people having connection issues know the reason...

Also EEE is mainly hardware related and I am not really sure if the chip used in Pi4/5 even allow such feature.

PS. Everything is fine here with a Cisco Catalyst switch, in 100 and 1000 modes.

@b4dpxl
Copy link
Author

b4dpxl commented Mar 28, 2024

In my case the problematic switch is a Netgear GS108Ev3, but a newer version Netgear GS108PEv3 is fine. I suspect most Pi users aren't running Cisco network equipment though 🤨

Well, as I understand it, the Pi4 and Pi5 both share the ethernet chip, and the Pi4 definitely allows you to disable EEE. It's referenced here in the overlays document and here is the pull request which was merged to allow it to be disabled via dtparam.

Clearly this was a known issue, otherwise this wouldn't have been added. As per the pull:

The EEE implementation on BCM2711 doesn't work well with all switches.

IMO it's not fair to place the blame on the switch when there are multiple reported switches at issue, and one common factor (the Pi). Did you actually look at any of the links I provided before?

@alessandromrc
Copy link

In my case the problematic switch is a Netgear GS108Ev3, but a newer version Netgear GS108PEv3 is fine. I suspect most Pi users aren't running Cisco network equipment though 🤨

Well, as I understand it, the Pi4 and Pi5 both share the ethernet chip, and the Pi4 definitely allows you to disable EEE. It's referenced here in the overlays document and here is the pull request which was merged to allow it to be disabled via dtparam.

Clearly this was a known issue, otherwise this wouldn't have been added. As per the pull:

The EEE implementation on BCM2711 doesn't work well with all switches.

IMO it's not fair to place the blame on the switch when there are multiple reported switches at issue, and one common factor (the Pi). Did you actually look at any of the links I provided before?

Yes, I've looked into the links you shared but not really deeply since I had not much time, anyways since you mentioned that it can be disabled on Pi4, I will later test out on a 4 and see the results I get and if there's something possible in the 5...

@alessandromrc
Copy link

Just tested on a 4 and I can see the NIC shutting down and powering back on... Will check later what could cause the problem on the 5.

@alessandromrc
Copy link

alessandromrc commented Mar 29, 2024

Turns out I've found difference between the Pi4 and the Pi5, if you run "lspci" you can see that on the Pi5 the ethernet controller is connected through PCI unlike the 4...

I think this is causing some issues to ethtool at this moment, will continue to look around.

PS. I think that the PCI ethernet controller is practically the new RP1 chip.

@alessandromrc
Copy link

Also @b4dpxl if you're interested into the RP1 chip you can find some infos here: https://datasheets.raspberrypi.com/rp1/rp1-peripherals.pdf

Sadly there's nothing about EEE so I guess we can do nothing at this moment : /

@alessandromrc
Copy link

After looking around the current implementation of Ethernet on RP1 doesn't support EEE at all since the implementation of it is included in IEEE 802.3az, meanwhile on the RP1 docs it is just IEEE 802.3...

Not sure if this will be changed in the future but I doubt anything will come soon, even because EEE is a good feature to always have enabled.

@pelwell
Copy link
Contributor

pelwell commented Apr 3, 2024

I have seen no mention of EEE or 802.3az in the documentation for the Ethernet MAC in RP1.

@pelwell pelwell closed this as completed Apr 3, 2024
@SensorAnalyticsAus
Copy link

SensorAnalyticsAus commented May 15, 2024

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:

scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed

Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.

Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

@alessandromrc
Copy link

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:

scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed

Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.

Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

@SensorAnalyticsAus
Copy link

SensorAnalyticsAus commented May 18, 2024

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   12GB 105.8MB/s   02:00


scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                        19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe

*** Connection stalled in just over 2 mins of data transfer

MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  41.5MB/s   22:00

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  42.8MB/s   21:20

*** Connection stable over 43 mins of data transfer

In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?

@alessandromrc
Copy link

alessandromrc commented May 18, 2024

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   12GB 105.8MB/s   02:00


scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                        19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe

*** Connection stalled in just over 2 mins of data transfer

MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  41.5MB/s   22:00

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  42.8MB/s   21:20

*** Connection stable over 43 mins of data transfer

In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?

I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...

Either ways, could you run the command:

uname -a

So we can at least understand what kernel you are running on, the output in my case is this one:
Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux

Just finished transferring 100GBs... I guess the problem is not happening here at all:

alessandro@alessandro-laptop:~$ scp [email protected]:/home/alessandro/test ./
[email protected]'s password: 
test                                                                                                                                                100%  100GB  93.6MB/s   18:14 

@SensorAnalyticsAus
Copy link

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   12GB 105.8MB/s   02:00


scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                        19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe

*** Connection stalled in just over 2 mins of data transfer

MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  41.5MB/s   22:00

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  42.8MB/s   21:20

*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?

I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...

Either ways, could you run the command:

uname -a

So we can at least understand what kernel you are running on, the output in my case is this one: Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux

Just finished transferring 100GBs... I guess the problem is not happening here at all:

alessandro@alessandro-laptop:~$ scp [email protected]:/home/alessandro/test ./
[email protected]'s password: 
test                                                                                                                                                100%  100GB  93.6MB/s   18:14 

That's good to hear. Problem occurs at my end only when using direct cable between PI5 and any other host, for some reason nmtui does not allow changing or even adding IPV4 for link-local connections or I would tried with IPV4 stack for comparison. PI5's uname:

uname -a
Linux rpi5 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

Perhaps repeating a file transfer with IPV6 src and dst addresses may reproduce the problem I am seeing.

@alessandromrc
Copy link

alessandromrc commented May 19, 2024

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   12GB 105.8MB/s   02:00


scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                        19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe

*** Connection stalled in just over 2 mins of data transfer

MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  41.5MB/s   22:00

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  42.8MB/s   21:20

*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?

I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...
Either ways, could you run the command:

uname -a

So we can at least understand what kernel you are running on, the output in my case is this one: Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux
Just finished transferring 100GBs... I guess the problem is not happening here at all:

alessandro@alessandro-laptop:~$ scp [email protected]:/home/alessandro/test ./
[email protected]'s password: 
test                                                                                                                                                100%  100GB  93.6MB/s   18:14 

That's good to hear. Problem occurs at my end only when using direct cable between PI5 and any other host, for some reason nmtui does not allow changing or even adding IPV4 for link-local connections or I would tried with IPV4 stack for comparison. PI5's uname:

uname -a
Linux rpi5 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

Perhaps repeating a file transfer with IPV6 src and dst addresses may reproduce the problem I am seeing.

I was running on kernel 6.22 so I will update my system to the latest kernel 6.30 and see if anything changes (I don't remember seeing any change on eth related stuff tho)

Just tested again and I confirm to have no issues with IPv4/6 at all when sharing my network using Debian.

Are you sure your network cable is functional, also if you run "ifconfig" do you see any Tx/Rx error on the eth interface? That would mean something is going really wrong, here's my output:

eno1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 08:bf:b8:d0:3c:d0  txqueuelen 1000  (Ethernet)
        RX packets 19174297  bytes 26668819810 (26.6 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2227528  bytes 160733910 (160.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

As you can see I've got no errors at all and the transfer went well (I stopped it at 26GB).

I could try on a MBP with M1 but I doubt the problem relies on MacOS.

@SensorAnalyticsAus
Copy link

Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.

I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.

I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)

scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   12GB 105.8MB/s   02:00


scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                        19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe

*** Connection stalled in just over 2 mins of data transfer

MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  41.5MB/s   22:00

scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar                                       100%   54GB  42.8MB/s   21:20

*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?

I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...
Either ways, could you run the command:

uname -a

So we can at least understand what kernel you are running on, the output in my case is this one: Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux
Just finished transferring 100GBs... I guess the problem is not happening here at all:

alessandro@alessandro-laptop:~$ scp [email protected]:/home/alessandro/test ./
[email protected]'s password: 
test                                                                                                                                                100%  100GB  93.6MB/s   18:14 

That's good to hear. Problem occurs at my end only when using direct cable between PI5 and any other host, for some reason nmtui does not allow changing or even adding IPV4 for link-local connections or I would tried with IPV4 stack for comparison. PI5's uname:

uname -a
Linux rpi5 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

Perhaps repeating a file transfer with IPV6 src and dst addresses may reproduce the problem I am seeing.

I was running on kernel 6.22 so I will update my system to the latest kernel 6.30 and see if anything changes (I don't remember seeing any change on eth related stuff tho)

Just tested again and I confirm to have no issues with IPv4/6 at all when sharing my network using Debian.

Are you sure your network cable is functional, also if you run "ifconfig" do you see any Tx/Rx error on the eth interface? That would mean something is going really wrong, here's my output:

eno1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 08:bf:b8:d0:3c:d0  txqueuelen 1000  (Ethernet)
        RX packets 19174297  bytes 26668819810 (26.6 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2227528  bytes 160733910 (160.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

As you can see I've got no errors at all and the transfer went well (I stopped it at 26GB).

I could try on a MBP with M1 but I doubt the problem relies on MacOS.

Great suggestions. I have tried multiple cables of various lengths and the tests are with a CAT6 cert cable. Also it matters not whether it's mbp of a Debian. This time I tested between PI5 and PI4.

ifconfig -a outputs for PI5 and PI4 before the start of file transfer
RPI5

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::e02a:17b4:c527:84e3  prefixlen 64  scopeid 0x20<link>
        ether 2c:cf:67:08:68:e9  txqueuelen 1000  (Ethernet)
        RX packets 65  bytes 11160 (10.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 109  bytes 20235 (19.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 112 

RPI4

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.59.177  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::6617:79bc:5bd6:5117  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:b7:b3:29  txqueuelen 1000  (Ethernet)
        RX packets 5465  bytes 889281 (868.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3156  bytes 355236 (346.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

SCP RPI5 -> RPI4 (Bullseye Linux rpi4c 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux)

scp -6 BIG.tar [fe80::6617:79bc:5bd6:5117%eth0]:/mnt/SSD
BIG.tar                                       100%   12GB  13.1MB/s   16:18

Now this is what really happened during this SCP session:
File transfer stalled at 50% after 2mins approx, PI5's eth0 lost its IPV6 address binding, a few mins later SCP file transfer restarts PI5's eth0 IPV6 binding appears restored, it then stalls again at 96% binding lost, file restarts again after few mins binding restored. Seems PI5's eth0 is regularly losing and regaining its IPV6 address binding. I will add few PI5's iconfig outputs to show these binding being lost-restored-lost phenomenon.

RPI5

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::e02a:17b4:c527:84e3  prefixlen 64  scopeid 0x20<link>
        ether 2c:cf:67:08:68:e9  txqueuelen 1000  (Ethernet)
        RX packets 65  bytes 11160 (10.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 109  bytes 20235 (19.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 112
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 2c:cf:67:08:68:e9  txqueuelen 1000  (Ethernet)
        RX packets 791970  bytes 86985777 (82.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9362558  bytes 13809932029 (12.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 112
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::e02a:17b4:c527:84e3  prefixlen 64  scopeid 0x20<link>
        ether 2c:cf:67:08:68:e9  txqueuelen 1000  (Ethernet)
        RX packets 823384  bytes 90279654 (86.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9667899  bytes 14259386311 (13.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 112

I then did a bash script to grep PI5's eth0 inet6 address every 15 secs, here are the outputs:

link-local-test.sh
 Mon 20 May 12:10:44 AEST 2024 inet6 up
Mon 20 May 12:10:59 AEST 2024 User down!!
Mon 20 May 12:11:14 AEST 2024 User down!!
Mon 20 May 12:11:29 AEST 2024 User down!!
Mon 20 May 12:11:44 AEST 2024 User down!!
Mon 20 May 12:11:59 AEST 2024 User down!!
Mon 20 May 12:12:14 AEST 2024 User down!!
Mon 20 May 12:12:29 AEST 2024 User down!!
Mon 20 May 12:12:44 AEST 2024 User down!!
Mon 20 May 12:12:59 AEST 2024 User down!!
Mon 20 May 12:13:14 AEST 2024 User down!!
Mon 20 May 12:13:29 AEST 2024 User down!!
Mon 20 May 12:13:44 AEST 2024 User down!!
Mon 20 May 12:13:59 AEST 2024 User down!!
Mon 20 May 12:14:14 AEST 2024 User down!!
Mon 20 May 12:14:29 AEST 2024 User down!!
Mon 20 May 12:14:44 AEST 2024 User down!!
Mon 20 May 12:14:59 AEST 2024 User down!!
Mon 20 May 12:15:14 AEST 2024 User down!!
Mon 20 May 12:15:29 AEST 2024 User down!!
Mon 20 May 12:15:44 AEST 2024 User down!!
 Mon 20 May 12:15:59 AEST 2024 inet6 up
 Mon 20 May 12:16:14 AEST 2024 inet6 up
 Mon 20 May 12:16:29 AEST 2024 inet6 up
 Mon 20 May 12:16:44 AEST 2024 inet6 up
 Mon 20 May 12:16:59 AEST 2024 inet6 up
 Mon 20 May 12:17:14 AEST 2024 inet6 up
 Mon 20 May 12:17:29 AEST 2024 inet6 up
 Mon 20 May 12:17:44 AEST 2024 inet6 up
 Mon 20 May 12:17:59 AEST 2024 inet6 up
 Mon 20 May 12:18:14 AEST 2024 inet6 up
 Mon 20 May 12:18:29 AEST 2024 inet6 up
 Mon 20 May 12:18:44 AEST 2024 inet6 up
Mon 20 May 12:18:59 AEST 2024 User down!!
Mon 20 May 12:19:14 AEST 2024 User down!!
Mon 20 May 12:19:29 AEST 2024 User down!!
Mon 20 May 12:19:44 AEST 2024 User down!!
Mon 20 May 12:19:59 AEST 2024 User down!!
Mon 20 May 12:20:14 AEST 2024 User down!!
Mon 20 May 12:20:29 AEST 2024 User down!!
Mon 20 May 12:20:44 AEST 2024 User down!!
Mon 20 May 12:20:59 AEST 2024 User down!!
Mon 20 May 12:21:14 AEST 2024 User down!!
Mon 20 May 12:21:29 AEST 2024 User down!!
Mon 20 May 12:21:44 AEST 2024 User down!!
Mon 20 May 12:21:59 AEST 2024 User down!!
Mon 20 May 12:22:14 AEST 2024 User down!!
Mon 20 May 12:22:29 AEST 2024 User down!!
Mon 20 May 12:22:44 AEST 2024 User down!!
Mon 20 May 12:22:59 AEST 2024 User down!!
Mon 20 May 12:23:14 AEST 2024 User down!!
Mon 20 May 12:23:29 AEST 2024 User down!!
Mon 20 May 12:23:44 AEST 2024 User down!!
 Mon 20 May 12:23:59 AEST 2024 inet6 up
 Mon 20 May 12:24:14 AEST 2024 inet6 up
...
...

PI5's eth0 IPV6 binding regularly flaps between 3 mins up and 5 mins down

@julienrobin28
Copy link

I have seen no mention of EEE or 802.3az in the documentation for the Ethernet MAC in RP1.

Hi there, the issue has been closed by @pelwell a little bit too fast, on a mistake: peers that are connected to the Pi 5 are showing that in reality, EEE is both active and enabled on the Pi 5 Ethernet MAC, even if not mentioned into its documentation.

Result from an EEE capable laptop directly connected to the Pi 5:

root@pop-crobe:/home/user# ethtool --show-eee enp4s0f1
EEE settings for enp4s0f1:
	EEE status: enabled - active
	Tx LPI: disabled
	Supported EEE link modes:  100baseT/Full
	                           1000baseT/Full
	Advertised EEE link modes:  100baseT/Full
	                            1000baseT/Full
	Link partner advertised EEE link modes:  100baseT/Full
	                                         1000baseT/Full

When this laptop is connected to a device or switch on which EEE is disabled or not supported, it says EEE status: enabled - inactive, so the output is reliable.

Just to confirm that disabling EEE is solving the link stability issue between the laptop and the Pi 5, from the Laptop:

root@pop-crobe:/home/user# ethtool --set-eee enp4s0f1 eee off
root@pop-crobe:/home/user# ethtool --show-eee enp4s0f1
EEE settings for enp4s0f1:
	EEE status: disabled
	Tx LPI: disabled
	Supported EEE link modes:  100baseT/Full
	                           1000baseT/Full
	Advertised EEE link modes:  Not reported
	Link partner advertised EEE link modes:  Not reported

And this solves the link stability issues with the Pi 5.

But when both my Laptop and the Pi 5 are connected to an EEE compliant switch (unmanaged switch), this can't be done from Pi 5 (and then in my case, link stability issue then occurs again, between the switch and the Pi 5).

On most of my Pi 4, there was the same stability issue in the same circumstances. However the ability to disable EEE from the Pi 4 allowed to retrieve a stable link everywhere. Today, we would clearly need it for the Pi 5 too.

Is it really needed?

I was surprised to see how much efforts were provided here to contradict and challenge the reality of the issue, instead of looking possible ways to just disable this. We probably shouldn't be spending so much efforts questioning the usefulness of a feature that has already been approved, used, and implemented in so much drivers for so long already.

Of course, EEE wasn't supposed to need disabling. Most of the time, we don't need to. To be honest, Pi 4 and 5 are the only devices on which I sometimes encounter link stability issues with EEE, and not even with all peers. Is this Pi 5 fault? Is this not? Let's say we aren't sure yet, because this doesn't matter anyway: the need to disable EEE from Pi 5 in some circumstances remains the same.

@go2sh
Copy link

go2sh commented Jul 23, 2024

Hey,
I stumpled across this issue while trying to setup a PTP connection between my laptop and a rpi5. The path delay was very long and this was due to EEE, which I disabled at my laptop. I also wanted to disable it at the rpi5 because the path delay is still long.

Regarding the EEE issue. Its always a mix between mac and phy. Lets sort it out a bit:
EEE is part of the auto-negotiation feature of the IEEE802.3. So its only enabled if both link partners support EEE. If you disable it at one link partner it should be disabled at both sides. Auto-negotiation is mainly a phy thing. The OS needs to read the results of the auto-neg and configure the mac accordingly.
How does EEE work?
EEE enables the phy/mac to disable the transmission and reception logic to save power. This is done by disabling the phy transmitter for a certain amount of a time and then send a refresh symbol to not loose the link. The receiving phy can disable the clock to the mac to save more power at the mac side. This mechanism works in both ways the same. (See for example https://ww1.microchip.com/downloads/en/DeviceDoc/00002841B.pdf)

How can this impact the link performance? If the transmitter is disabled from time to time two things can happen: a) Packets will be delayed until the transmitter left low power mode (my problem with PTP) or the receiving phy interprets the disabled transmitter as link-down. (Seams like the problem in this ticket).

As a result, if its true that the Cadence IP doesn't support EEE, the phy should not advertise it.
The cadence IP does support EEE, but it doesn't seam to be used by the linux driver. The phy supports it. So, I guess there is an issue between the mac driver, phy driver and phy handling.

@julienrobin28
Copy link

Hi @go2sh,

As you explained, I can confirm once you succeeded to disable EEE at one side, it becomes inactive on both (from Linux machines that are able to report or change EEE status, you can use ethtool --show-eee IFNAME to see if active or not). When you set ethtool --set-eee IFNAME eee off from one side, the other side report (if able to do so) that it's now inactive, and even if unable to report EEE status, if any issue was occurring because of EEE, related issue should have disappeared (this is what's happening on Pi 4 and 5 at least).

What may be improved there, is that macb driver about cadence MAC IP (used by Pi 5's RP1 chip) is not able to change/report the EEE status. The partial documentation of this Cadence MAC IP says Support for 802.3az Energy Efficient Ethernet, and there's a ton of settings registers in there, but the precise definition of each bit isn't provided to the public (Raspberry Pi engineers may have access to it ?), hoping some of them may be used to add EEE status handling to the macb driver.

For the original cause of the link stability issue when EEE is active, I'm 99.9% the issue isn't due to a faulty driver or faulty Cadence MAC IP, because the only other device on which I encountered the same issue is Raspberry Pi 4, which (despite using another GbE controller and driver) shares the same BCM54213PE chip and physical/electrical stuff around it for the RJ45 port (which may be the common and problematic stuff there). However Pi 4 had the ability to disable EEE itself thanks to its different associated driver (using ethtool or by adding dtparam=eee=off into config.txt).

PS:

How can this impact the link performance? If the transmitter is disabled from time to time two things can happen: a) Packets will be delayed until the transmitter left low power mode (my problem with PTP) or the receiving phy interprets the disabled transmitter as link-down. (Seams like the problem in this ticket).

This looks convincing 👍 had some packets drops too (sometimes without having eth port reported disconnected and reconnected), so I suspect that in low power mode, being on Pi 4 or 5, for some reason the BCM54213PE chip isn't working as good as expected, and causes the misfires, despite being advertised EEE compatible. Is this because of voltage, capacitive, inductive or resistive issues (or something else), I don't really know.

@go2sh
Copy link

go2sh commented Jul 24, 2024

I have access to the complete cadence documentation. But you can also check other chips like the traveo 2 or xilinx socs, which internally use gem as well.
The gem supports sending lpi patterns, but no timer or things like this, and receiving lpi patterns signal this to the driver.

Unfortunately, I have no access to the broadcom phy. I'm really interested also with regards to the internal delay for accurate ptp delay measurement.

@oliverkrystal
Copy link

I have pi zero's, pi 3, and pi 4s with this eee issue.

@pelwell
Copy link
Contributor

pelwell commented Apr 10, 2025

The Pi 3 Ethernet does not support EEE, and it should never be enabled by the switch. Pi Zero doesn't have an Ethernet interface.

@oliverkrystal
Copy link

The Pi 3 Ethernet does not support EEE, and it should never be enabled by the switch. Pi Zero doesn't have an Ethernet interface.

Good to know I can take those things of the list of items to fiddle with. I suppose the correct thing to do would be to start a new issue. This issue with the Pi Zero I solved by having /sbin/iw wlan0 set power_save run at boot.

The Pi 3 I'm not sure of the issue. It's a headless system running it a shed that drops every week or so directly connected to ethernet. I added a big capacitor, that did nothing to help stability. Sometimes it randomly comes back online. I added a UPS figuring the power was dropping and that didn't help either. I experienced this issue with pi-star and WPSD. Swapping OSs is not an option given the purpose of use.

@alessandromrc
Copy link

The Pi 3 Ethernet does not support EEE, and it should never be enabled by the switch. Pi Zero doesn't have an Ethernet interface.

Good to know I can take those things of the list of items to fiddle with. I suppose the correct thing to do would be to start a new issue. This issue with the Pi Zero I solved by having /sbin/iw wlan0 set power_save run at boot.

The Pi 3 I'm not sure of the issue. It's a headless system running it a shed that drops every week or so directly connected to ethernet. I added a big capacitor, that did nothing to help stability. Sometimes it randomly comes back online. I added a UPS figuring the power was dropping and that didn't help either. I experienced this issue with pi-star and WPSD. Swapping OSs is not an option given the purpose of use.

Doubt it's a software issue tho... if you solved it by setting "power_save" run at boot... maybe it might be the actual regulator on-board failing? For how long do you have this PI, did it always have this issue since you got it?

@oliverkrystal
Copy link

Doubt it's a software issue tho... if you solved it by setting "power_save" run at boot... maybe it might be the actual regulator on-board failing? For how long do you have this PI, did it always have this issue since you got it?

The pi zero had this issue. Pretty sure it's always had a wireless stability issue. I found that idea on another forum so it's either a unknown consistent problem or a unacknowledged problem.

The pi3 I rebooted this morning and it came right back to life. It sucks because it's headless and remote and takes time to fault out. At this point I'm going to "solve" the issue by forcing a reboot every morning at 2:25 AM. But I'd sure as heck like to actually solve the problem instead of just hacking it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants