Unplug of ethernet cable is not detected

[polomsky]

Describe the bug
If RPI is started with unplugged ethernet cable, after full boot it correctly detects that ethernet cable was connected or disconnected, e.g. /sys/class/net/eth0/carrier is changing content.
If RPI is started with plugged ethernet cable, it always responds that cable is connected.

To reproduce

1. turn on device with ethernet cable plugged in (ethernet must work)
2. watch /sys/class/net/eth0/carrier
3. unplug ethernet

Expected behaviour
/sys/class/net/eth0/carrier content should be 0

Actual behaviour
unplug of ethernet is not detected but on ethernet card yellow light is switched off.
/sys/class/net/eth0/carrier content is 1

System

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
  RPI3 model B+
• Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2019-04-08
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, e910bf55ac92e27c6e77043e8995bcb5e0a2d742, stage2

• Which firmware version (vcgencmd version)?

Mar 27 2019 15:45:53 
Copyright (c) 2012 Broadcom
version 2e98b31d18547962e564bdf88e57b3df7085c29b (clean) (release) (start)

• Which kernel version (uname -a)?

Linux raspberrypi 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

Logs
If applicable, add the relevant output from dmesg or similar.

Additional context
Add any other relevant context for the problem.

[XECDesign]

Also reported here:
raspberrypi/firmware#1100

[sudhamani-hcl]

Hi Team,

I am not able to replicate the issue with the following system information. Update me if I am missing anything.

System:

1. RPI3 model B+
2. OS version
   cat /etc/rpi-issue
   Raspberry Pi reference 2019-04-08
   Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, e910bf55ac92e27c6e77043e8995bcb5e0a2d742, stage4
3. Kernel version
   Linux raspberrypi 4.14.98-v7+ Add FBUNSUPPORTED ioctl for bcm2708_fb #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

Thanks & Regards,
Sudhamani B

[JamesH65]

It would be interesting to know if the problem is showing for anyone using the latest kernel (4.19)

[sudhamani-hcl]

Hi Team, I would like to know does anyone working on this issue? Regards, Sudhamani.

[pelwell]

It would be interesting to know if the problem is showing for anyone using the latest kernel (4.19)

I would like to know does anyone working on this issue?

I think @JamesH65 answered your question before you asked it - we don't work on problems unless we know they exist.

[XECDesign]

The problem is still there with 4.19.58

[jasonemann]

I'm seeing this problem on my Raspberry Pi 3 B+ as well. I noticed it when power cycling a small switch with four RPi's (three 3B's, one 3B+) connected caused all but the RPi3+ to request a new DHCP lease (as was the intent).

root@rpi3p-01:~# uname -a
Linux rpi3p-01 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

root@rpi3p-01:~# vcgencmd version
Aug 15 2019 12:07:10
Copyright (c) 2012 Broadcom
version 0e6daa5106dd4164474616408e0dc24f997ffcf3 (clean) (release) (start_cd)

(Booted with cable connected)
root@rpi3p-01:~# cat /sys/class/net/eth0/carrier
1

(Disconnected cable)
root@rpi3p-01:~# cat /sys/class/net/eth0/carrier
1

(Connected cable)
root@rpi3p-01:~# cat /sys/class/net/eth0/carrier
1

root@rpi3p-01:~# journalctl | grep carrier | grep eth0
Aug 23 14:15:02 rpi3p-01 dhcpcd[348]: eth0: waiting for carrier
Aug 23 14:15:03 rpi3p-01 dhcpcd[348]: eth0: carrier acquired

(Booted with cable disconnected)
root@rpi3p-01:~# cat /sys/class/net/eth0/carrier
0

(Connected cable)
root@rpi3p-01:~# cat /sys/class/net/eth0/carrier
1

(Disconnected cable)
root@rpi3p-01:~# cat /sys/class/net/eth0/carrier
0

(Connected cable)
root@rpi3p-01:~# cat /sys/class/net/eth0/carrier
1

root@rpi3p-01:~# journalctl | grep carrier | grep eth0
Aug 23 14:20:12 rpi3p-01 dhcpcd[325]: eth0: waiting for carrier
Aug 23 14:26:44 rpi3p-01 dhcpcd[432]: eth0: carrier acquired
Aug 23 14:27:47 rpi3p-01 dhcpcd[432]: eth0: carrier lost
Aug 23 14:27:56 rpi3p-01 dhcpcd[432]: eth0: carrier acquired

[sudhamani-hcl]

Hi,

This problem is occurring so rarely on the following kernel
Linux raspberrypi 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

Anyone working on this isse?

Regards,
Sudhamani

[brubbel]

Still present in 4.19.93-v7+
Workaround is to request the carrier state directly from the LAN78xx (using ETHTOOL_GLINK). If a discrepancy between the state and the carrier flag is detected, send USBDEVFS_RESET to the driver. From then on, it starts working normally.

[JamesH65]

@brubbel Have you got a patch that shows that change in the driver?

[paul-1]

@JamesH65 has anyone been able to resolve this issue. This is still a problem on 5.4 kernels.

[brubbel]

FWIW, I did not patch the driver but use a daemon script to detect this and restart the driver. Never seen the issue since then (on quite a lot of RPI's), so I don't know if the root problem has been resolved or not.

Poll LAN controller link status:

import fcntl
import socket
import array
import struct

class Ethtool(object):
    # https://github.com/torvalds/linux/blob/master/include/uapi/linux/sockios.h
    SIOCETHTOOL = 0x8946
    # https://github.com/torvalds/linux/blob/master/include/uapi/linux/ethtool.h
    ETHTOOL_GLINK = 0x00000000a

    @staticmethod
    def link_detected(ifname):
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        ecmd = array.array('B', struct.pack('2I', Ethtool.ETHTOOL_GLINK, 0))
        args = struct.pack('16sP', ifname, ecmd.buffer_info()[0])
        fcntl.ioctl(sock, Ethtool.SIOCETHTOOL, args)
        res = ecmd.tostring()
        return bool(struct.unpack('4xI', res)[0])

Then compare with cat /sys/class/net/ifaceNameHere/carrier

If those two disagree, reset LAN controller:

import os
import re
import fcntl
import logging

from subprocess import Popen
from subprocess import PIPE

class UsbDevFS(object):
    LOGGER = logging.getLogger('UsbDevFS')
    LAN78XX_ID = '0424:7800'
    PATTERN_LSUSB = re.compile(r'Bus\s(\d{3})\sDevice\s(\d{3}):', re.S)
    # https://github.com/torvalds/linux/blob/master/include/uapi/linux/usbdevice_fs.h
    USBDEVFS_RESET = ord('U') << (4*2) | 20

    @classmethod
    def reset_lan78xx(cls):
        try:
            path = cls._get_lan78xx_path()
        except LookupError:
            return  # LAN78XX not found
        fd = os.open(path, os.O_WRONLY)
        try:
            fcntl.ioctl(fd, UsbDevFS.USBDEVFS_RESET, 0)
        finally:
            os.close(fd)

    @classmethod
    def _get_lan78xx_path(cls):
        cmd = ['lsusb', '-d', cls.LAN78XX_ID]
        process = Popen(cmd, stdout=PIPE, stderr=PIPE)
        try:
            (stdout, stderr) = process.communicate(timeout=15)
        except Exception:
            process.kill()
            raise
        if process.returncode != 0:
            cls.LOGGER.error('Command failed: {:s}'.format(' '.join(cmd)))
        m = re.search(cls.PATTERN_LSUSB, stdout.decode('ascii'))
        if not m:
            cls.LOGGER.error(
                'LAN78XX device id={:s} not found'.format(cls.LAN78XX_ID)
            )
            raise LookupError
        bus = m.group(1)
        device = m.group(2)
        return '/dev/bus/usb/{:s}/{:s}'.format(bus, device)

[pelwell]

This is just a quick update to say that I can reproduce the problem and am looking for the right solution. As a workaround you could try putting "ethtool -d eth0" in /etc/rc.local (or running "sudo ethtool -d eth0" manually), and see if the carrier tracking is correct from then on, or if it reverts to the bad state after a while.

[brubbel]

If I remember correctly this occurs only once. After reset of the LAN78XX device all goes well.

[pelwell]

Does that mean you will try my workaround instead of your script?

[paul-1]

It helps until the interface is taken down.

Boot system with cable in.
sudo ethtool -d eth0
Remove Cable - Carrier goes to 0
Plugin cable - carrier goes to one

However

Remove Cable
ifconfig eth0 down
ifconfig eth0 up
plug cable in

/sys/class/net/eth0/carrier never goes back to a "1" And the interface is dead until it is reset. When resetting the USB interface, it has to happen when the cable is unplugged for it to work properly.

[pelwell]

Thanks - that's a useful test case.

[pelwell]

The carrier status disconnect is caused by the loss of interrupts from the PHY, which in turn is caused by resetting the MAC's view of the PHY interrupt status while an interrupt is pending. Until the lan78xx driver sees the interrupt status bit in the MAC it won't forward the interrupt to the PHY driver, and until it does that no further PHY interrupts will be generated.

sudo ethtool -d eth0 breaks the deadlock by reading the PHY's interrupt status, but a proper solution is to explicitly acknowledge the PHY interrupts after clearing the bit in the MAC.

The fix is in rpi-5.4.y and rpi-5.10.y, and will appear in future kernel builds.

[juergh]

Duplicates of this bug:
raspberrypi/firmware#1100
#3939

[pelwell]

The patch mentioned above is in the current rpi-update kernel, which has just switched to rpi-5.10.y.

[paul-1]

Tested with Kernel built from latest 5.4.83, everything looks to be operating correctly.

Thanks.