RPI 3B+ state of /sys/class/net/eth0/carrier stuck at 1

[brubbel]

TLDR; The eth0 carrier state flag gets stuck at '1' if /sys/class/net/eth0/carrier is polled within <1 second after unplugging the network cable. Systems that depend on it then fail (e.g. route).

Hardware (3 devices were tested, on 2 different networks):

1. RPI3B+ (rev. a020d3), new raspbian (kernel 4.14.79-v7+).
2. Raspberry pi official power supply (5.1V, 2.5A)
3. peripherals: ethernet cable, usb keyboard (dell), hdmi screen (dell).

Setup

1. OS (2018-11-13-raspbian-stretch-lite.img) is written to SD-card, ssh is not enabled.
2. ethernet cable is plugged in before boot.
3. local login.
4. cat /sys/class/net/eth0/carrier (do enter yet!)
5. unplug ethernet cable
6. enter within 1 second after unplug.
7. (To retry a reboot is needed.)

Result
The /sys/class/net/eth0/carrier flag gets stuck at '1'.
Also, the 'route' command hangs as the os probably doesn't know the interface is down (route -n works).

Expected result
The /sys/class/net/eth0/carrier flag becomes '0'

Situation where this issue is relevant
A script may be checking the carrier flag to failover to WiFi, for example every 10 seconds. However, every once in a while a race condition can occur where this bug is triggered on disconnecting the cable.

[brubbel]

@JamesH65 Very possibly related:
raspberrypi/linux#2617 (comment)
raspberrypi/linux#2614 (comment)

[XECDesign]

This is also affecting dhcpcd, which fails to switch over to wifi if the cable is unplugged.

In my case, I can reproduce this by booting with the ethernet cable plugged in and running ip monitor link dev eth0. Then unplugging or plugging the cable back in shows no activity.

If I boot with the cable unplugged, it works as expected.

The issue is present in 4.14 and 4.19.

I tried a patch that adds tasklet_schedule(&dev->bh); back into lan78xx_open, but that made no difference.

[brubbel]

One remarkable thing is that ETHTOOL_GLINK (SIOCETHTOOL ioctl) reports the correct link status, so it must be that the carrier status flag is not extracted from what the hardware reports in real time.
I suppose that the driver buffers the carrier state for performance reasons.

e.g. ethtool eth0

[XECDesign]

When you run ethtool, it reads the appropriate register, but only reports the value without tracking any changes. The driver has a different code path to handle carrier state changes, which is where the issue is. I haven't looked into how it works, but it's either an interrupt or polling mechanism that doesn't get triggered.

[JamesH65]

Looking at https://github.com/raspberrypi/linux/blob/rpi-4.14.y/drivers/net/usb/lan78xx.c#L1179, this is the section of code that handles the link being dropped, I wonder if there needs to be some sort of flushing of the various SKB's etc that might be in flight. For example, there was a recent fix to the end of this function https://github.com/raspberrypi/linux/blob/rpi-4.14.y/drivers/net/usb/lan78xx.c#L1237 that makes sure requried processing is done when the link comes up. Complete guess though.

What i don;t understand is why the polling seems to kill it, that's just a read as far as I cna see, so would not expect that to cause this. Unless there's a mutex issue somewhere perhaps?

[JamesH65]

I don't seem to be able to replicate this on 4.14.90 using any of the mechanisms above. Don't think there is anything unusual in my kernel setup, non-tainted, so a pure 4.14.90.

[brubbel]

Bug is still there. Just tested with 4.14.90-v7+ (upgraded from 4.14.79-v7+).
Link speed is 1000Mb/s when connected, if this may be relevant.

When cable is unplugged:

Linux raspberrypi 4.14.90-v7+ #1183 SMP Fri Dec 21 14:03:50 GMT 2018 armv7l GNU/Linux

pi@raspberrypi:~ $ sudo ethtool eth0
Settings for eth0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                             100baseT/Half 100baseT/Full 
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbag
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no

pi@raspberrypi:~ $ cat /sys/class/net/eth0/carrier
1

[JamesH65]

Odd. Not sure why I am not seeing it then. Tried the cat /sys/class/net/eth0/carrier as soon as I removed the ethernet cable, along with @XECDesign ip monitor test, and all worked fine. I'll try again.

[brubbel]

Strange. Since the rpi-update (and downgrade to original kernel), carrier is always stuck at 1, unless I boot without cable. Will check a second RPI in a moment.

[brubbel]

Second RPI (rpi-update to 4.14.94-v7+), cable is unplugged.

cat /sys/class/net/eth0/carrier
switches between
cat: /sys/class/net/eth0/carrier: Invalid argument
and
1

[brubbel]

More info (back to fresh SD card). I have the impression that thing became worse after the rpi-update.

Boot with cable: carrier stuck (only unbind/bind the driver solves it, and it stays solved)
Boot without cable: works! (carrier switches nicely as it should do)

pi@raspberrypi:~ $ cat /etc/rpi-issue 
Raspberry Pi reference 2018-11-13
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 7e0c786c641ba15990b5662f092c106beed40c9f, stage2

pi@raspberrypi:~ $ vcgencmd version
Nov  4 2018 16:31:07 
Copyright (c) 2012 Broadcom
version ed5baf9520a3c4ca82ba38594b898f0c0446da66 (clean) (release)

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux

[XECDesign]

I doubt it makes a difference, but as a data point - I have wifi connected at the same time. I could also give you my sd card tomorrow if it's not reproducible on your setup.

[brubbel]

Just tested: the desktop version (2018-11-13-raspbian-stretch.img) does not show the problem.
The lite version (2018-11-13-raspbian-stretch-lite.img) does.

I have the impression that 'something' is happening too fast after boot in the lite version, which can explain why an unbind/bind cycle solves the issue.
Could it be that the LAN7515 chip is not (completely) ready yet before the kernel initiates communication?

[XECDesign]

I was using the desktop version.

[brubbel]

I suppose it isn't possible to recompile/insmod the driver without recompiling the kernel?