Ethernet not working until ping is sent from the Pi (Only happens on Pi3b+)

[vanfanel]

Hi there,

I am not using dhcpcd. I am manually setting up the wired networking as I have been doing for years, but I have just moved from Pi3b to Pi3b+ and I found that I can't ping the Pi3b+ from the PC unless I ping the PC from the pi3b+ first. Strange!

Just in case, this is the simple sequence of commands I am using on my personal networking setup script:

sudo ip link set eth0 up
sudo ip addr add 192.168.5.8/24 dev eth0
sudo ip route add default via 192.168.5.7

Any ideas on what's going on?

[maxnet]

Probably because the driver doesn't start polling for incoming packets until an outbound one has been sent.
If there isn't anything you want to do that generates an outbound packet (e.g. synchronize your time with ntp), just arping your gateway IP in your networking setup script as workaround.

Think driver needs a tasklet_schedule(&dev->bh); added somewhere after link goes up.
Otherwise the tasklet (that handles both sends and receives) will not be run until something gets sent.

[lategoodbye]

@vanfanel I'm not able to reproduce your issue with Rasbian and Kernel 4.14.54 (downstream) and 4.18rc3 (upstream).

I added the following to /etc/dhcpcd.conf:
denyinterfaces eth0

And this to /etc/network/interfaces:
iface eth0 inet manual

Then i bring up the interface manual with the commands like in your script.

So please provide your changes more in detail. Also the output of ifconfig eth0 while the RPi 3B+ is unreachable would be helpful.

[vanfanel]

@maxnet : Thanks to your comment, I may have an idea on what's going on: since most people run Raspbian with services that are sending packets from the Pi3b+ at startup and I don't, this problem may have gone unnoticed to most people.

@lategoodbye : dhcpcd is NOT running on my system. There are no other services sending packets from the Pi. This is my list of services run at startup:

pi@raspberrypi:~ $ sudo systemd-analyze blame
          1.060s dev-mmcblk0p2.device
           724ms rc-local.service
           314ms systemd-udevd.service
           301ms systemd-udev-trigger.service
           288ms systemd-remount-fs.service
           177ms fake-hwclock.service
            89ms alsa-restore.service
            65ms sys-kernel-config.mount
            63ms dropbear.service
            40ms systemd-modules-load.service
            37ms boot.mount
            34ms systemd-fsck@dev-disk-by\x2dpartuuid-f5be54b9\x2d01.service

This is all my system needs. So the first difference about your system and mine is that you could have something sending some packets outside the Pi after you bring up the interface and configure it, while I don't.

Now, this is the output of ifconfig eth0 before I send out a PING from the Pi3b+ to the outside world:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.8  netmask 255.255.255.0  broadcast 0.0.0.0
        ether b8:27:eb:c4:06:13  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The IP is correct, as is the mask.

[maxnet]

Does is reproducible to me.

• Add ipv6.disable=1 to cmdline.txt

sudo bash
systemctl disable dhcpcd
systemctl disable avahi-daemon
systemctl disable systemd-timesyncd
reboot

Now configure IP manually.

[pelwell]

Yes, I see the failure with that configuration. A tcpdump in one terminal window on the Pi recording all activity sees no traffic until the very first packet is transmitted, at which point incoming traffic appears.

[JamesH65]

I suspect from my limited reading of the source, that there is a missing netif_wake_queue. There is one in lan78xx_rx_bh which kicks the queue, I presume if something is received, there's another in lan78xx_delayedwork, I think only activated if the EVENT_TX_HALT bit is set, but I wonder if there needs to be one somewhere else just to kick everything in to life on first startup. Or perhaps the EVENT_TX_HALT bit needs setting somewhere on start?

[maxnet]

I suspect from my limited reading of the source, that there is a missing netif_wake_queue.

I don't have time to compile a kernel today myself, so haven't tried it.
But before you start thinking too complicated, one simple suggestion:

Try if it helps if you insert a tasklet_schedule(&dev->bh); somewhere in the code after the link goes up.
E.g. here: https://github.com/raspberrypi/linux/blob/rpi-4.14.y/drivers/net/usb/lan78xx.c#L1224

As long as the tasklet is started once, it will reschedule itself again and again afterwards.
Just doesn't happen without a transmit right now.
Tasklet is responsible for both sending and polling/receiving. So if it doesn't run there will not be much action.

[lategoodbye]

You are absolutely right. usbnet ( used by smsc95xx ) already does this since 2.6.12, which really scares me because both drivers seems to have the same roots. Until the driver is ported to usbnet, the lan78xx should do the same and schedule the tasklet from the net device open callback.

[lategoodbye]

Btw it looks like that the root cause of this issue is the same as in #2554 and #2566 .

[maxnet]

schedule the tasklet from the net device open callback.

Wouldn't the moment the link is actually up (it has carrier), instead of when device is opened, be better?
So that it does not poll for data packets, if there is no cable connected.
(have the impression it polls quite aggressively, so if that can be avoided if user is using wifi that would be better)

[lategoodbye]

I think improving the polling should be done with an different patch, because the polling will be still aggressive if we disconnect the cable after the tasklet has been scheduled once. It's also important to keep the driver easy to read.

[maxnet]

Ok, fair enough.

For later reference: this is how usbnet prevents polling when link is down: 4b49f58
Leaves 10% more bandwidth for other USB devices if cable is not connected in their tests.

[pelwell]

That's definitely worth having - both the extra bandwidth and likely power saving.

[JamesH65]

I agree this is scary - as I referred in another issue, this should really use usbnet rather than hacking out great chucks of it's code which then never get the usbnet bug fixes.

[JamesH65]

Having a play incorproating the polling commit referenced above. Not trivial, having to guess where to put some stuff, as usbnet has diverged significantly in some places, but seem to start up, however, pulling the ethernet cable and it doesn't realise it's gone!

[lategoodbye]

As i said the polling behavior is a different issue, so we better don't mix them here. Instead of trying to backport features from usbnet it would be better to make the driver use usbnet. I think a simple cherry-pick won't work.

Back to the issue here, i'm waiting from feedback from @vanfanel and @dgriswo until i send the patch upstream.

[JamesH65]

Although just tested with a Top of tree kernel, and that shows the same symptoms. Unplug the ethernet plug, network icon top right stays on, ifconfig still indicates connected. Hmm.

[JamesH65]

@lategoodbye Definitely not a cherry pick - the differences are considerable. The effort to move it to usbnet is also, I suspect, considerable.

Can you try pulling the ethernet cable and seeing what happens? I suspect we may have introduced a regression (unless this has never operated correctly).

[lategoodbye]

I currently don't have access to my Pi 3B+ so you have to wait until the evening.

[lategoodbye]

@JamesH65 Thanks for spotting this. I can confirm that the link level isn't properly update as long as the dhcpcd is disabled. I reverted my last 2 patches and the symptom stays the same. So this isn't a regression, it never worked before :-(

Maybe we should open a new issue for this. Unfortunately i don't have the time for fixing this.

[maxnet]

Isn't that how it is supposed to work?
The network manager (be it dhcpcd or some other program) polling /sys/class/net/eth0/carrier or through ioctl and bringing down interface if carrier is lost, rather than that happening automatically?

[lategoodbye]

I would expect that the Ethernet LED state and /sys/class/net/eth0/carrier are in sync (indepented from dhcpcd is running or not). If i disconnect the cable, Ethernet LED goes off but the carrier is still 1.

[JamesH65]

This appears to be an issue with this commit #2614, . In my tests I reverted that commit, functionality came back, put it back in, functionality broke.

I have no idea why that might be - seems to be entirely unrelated to the effect being seen. Only thing that vaguely comes to mind is that if something 'underneath' during the queue walk tries to grab that spinlock, find it cannot, and gives up. Shoudl probably continue this discussion on the PR.

[lategoodbye]

I cannot confirm your observation. I reverted this patch, too and the issue is still there. This patch has nothing to do with link level handling.

[maxnet]

I would expect that the Ethernet LED state and /sys/class/net/eth0/carrier are in sync (indepented from
dhcpcd is running or not). If i disconnect the cable, Ethernet LED goes off but the carrier is still 1.

Does go to 0 for me.

This appears to be an issue with this commit #2614

Do have that commit in my build.
But not the new one covering the issue in this thread.

(As far as the icon in Raspbian goes, be aware it's a dhcpcd frontend, so don't look at that if dhcpcd is not running)

[lategoodbye]

@maxnet Do you have the same configuration as described to reproduce the original issue (dhcpcd off, manual and static ip configuration)?

[pelwell]

I see the issue even with dhcpcd running, and it's this commit (not the skb locking fix) that makes the difference.

This one-liner shows the problem:

watch -n 1 "ethtool eth0; cat /sys/class/net/eth0/carrier; ifconfig eth0"

In the failure case, only the output from ethtool changes when the cable is unplugged.

[maxnet]

@maxnet Do you have the same configuration as described to reproduce the original issue (dhcpcd off,
manual and static ip configuration)?

Yes
But like I said, I only have SKB lock fix, not the patch from this issue.

pi@raspberrypi:~$ uname -a
Linux raspberrypi 4.14.54v7-aufs #1 SMP Sun Jul 15 01:08:13 CEST 2018 armv7l GNU/Linux
pi@raspberrypi:~$ ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether b8:27:eb:42:9d:d6  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
pi@raspberrypi:~$ sudo ip link set eth0 up
pi@raspberrypi:~$ sudo ip addr add 192.168.178.111/24 dev eth0
pi@raspberrypi:~$ sudo ip route add default via 192.168.178.1
pi@raspberrypi:~$ ping www.google.com
PING www.google.com (172.217.17.68) 56(84) bytes of data.
64 bytes from ams16s30-in-f4.1e100.net (172.217.17.68): icmp_seq=1 ttl=54 time=21.8 ms
64 bytes from ams16s30-in-f4.1e100.net (172.217.17.68): icmp_seq=2 ttl=54 time=12.6 ms
^C
--- www.google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.663/17.237/21.811/4.574 ms
pi@raspberrypi:~$ cat /sys/class/net/eth0/carrier
1

Unplug my cable

pi@raspberrypi:~$ cat /sys/class/net/eth0/carrier
0