Dropped RX packets - RPi 1, 2, and 3

[stamster]

Hello,
I have multiple RPi's running on two locations. I noticed that each of them report dropped packets, but only in RX direction.

RPi 3:

eth0      Link encap:Ethernet  HWaddr b8:27:eb:xx:xx:xx  
          inet addr:192.168.100.20  Bcast:192.168.100.255  Mask:255.255.255.0
          inet6 addr: fe80::906e:f3:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:275242 errors:0 dropped:15336 overruns:0 frame:0
          TX packets:71301 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:38165404 (36.3 MiB)  TX bytes:23844353 (22.7 MiB)

RPi 2:
Just rebooted after latest rpi-firmware patch, and only 4 mins of uptime getting: RX packets:689 errors:0 dropped:5 overruns:0 frame:0

RPi 1 model B:
RX packets:647 errors:0 dropped:28 overruns:0 frame:0

RPi 1:
RX packets:209844 errors:0 dropped:14892 overruns:0 frame:0

The only single RPi which does not seem to be affected by this issue is another RPi 1 model B, where I didn't upgraded firmware.
So if it's a firmware related issue, the last known good firmware/kernel w/o RX drops is this one:

4.1.19+ #858 Tue Mar 15 15:52:03 GMT 2016 armv6l GNU/Linux

Mar 15 2016 14:48:20 
Copyright (c) 2012 Broadcom
version 1bf9a9a77026af9128a339c82d72e331d3532ee4 (clean) (release)

10 days uptime, and not a single drop:

eth0      Link encap:Ethernet  HWaddr b8:27:eb:xx:xx:xx  
          inet addr:192.168.100.30  Bcast:192.168.100.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1335268 errors:0 dropped:0 overruns:0 frame:0
          TX packets:721742 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:81649608 (77.8 MiB)  TX bytes:185224562 (176.6 MiB)

RPi's behave exactly the same on other networks, so it's not LAN related, since other PC's and devices do not have any drops with uptime of 250 days.

[JamesH65]

Any relevant messages in syslog? Also, any ideas when this problems started to occur?

[JamesH65]

A quick glance and some minor debugging at the driver level seems to indicate that it is not the driver itself that is dropping the packet - the code that increments the rx_dropped counter is not called. So presumably somewhere else in the stack is deciding to drop the packet. I'm not sure how to find out where in the stack though! Will investigate further.

[JamesH65]

It's quite possible this is harmless. The dropped packet count is not only used for errors, but also flagging up packets that are dropped because the linux stack doesn't handle them (some IPv6 stuff if you don't have IPv6 enabled for example). This might be one of those messages. I will try and find out which it is.

[stamster]

It's not IPv6 related as first thing I usually do after installation is disabling IPv6 on all interfaces via sysctl.conf kernel parameter. Is it harmless? You never really know, dropped packets shouldn't be there for sure. 19.05.2017. 17.11, "James Hughes" <notifications@github.com> је написао/ла:

…

It's quite possible this is harmless. The dropped packet count is not only used for errors, but also flagging up packets that are dropped because the linux stack doesn't handle them (some IPv6 stuff if you don't have IPv6 enabled for example). This might be one of those messages. I will try and find out which it is. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1954 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMNhf-Eu7da9v0nABpFCHCfdfwgiJGI5ks5r7bEhgaJpZM4M33E8> .

[JamesH65]

OK, not IPv6, but that was only given as an example of some packets that are simply discarded by the network stack. Dropped packets are not, necessarily, an error. If the stack gets a correctly formatted packet it doesn't implement, then it is dropped. It's not an error as such. So it is perfectly reasonable to be getting dropped packets and not worry about them. However, in this case I would like to know whether this is the case here.

[JamesH65]

Here is some (probably quite old) text on why a packet may be dropped.

Also worth noting that even if IPv6 is disable on the Pi, it could still receive IPv6 packets from other devices on the network, which will cause dropped packets

Softnet backlog full -- (Measured from /proc/net/softnet_stat)

Bad / Unintended VLAN tags

Unknown / Unregistered protocols

IPv6 frames when the server is not configured for IPv6

If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

[JamesH65]

After much faffing, I have build a utility I found called dropwatch, rebuilt the kernel to turn on a particular form of net logging and now have a list of addresses where packets are being dropped. However, cross references those addresses to the kernel isn't giving valid results, so i suspect all the drops are in modules.

What is worth noting is that the ifconfig dropped packets counter is a small subset of the actual number of packets dropped. I can see which address in my list is causing the ifconfigs drops, just need to figure out what code corresponds to that location.

[JamesH65]

OK, thanks to a timely intervention from @pelwell I do have some idea of the code that is dropping the packets. The ifconfig dropped packet counter appears to be incremented in the __netif_receive_skb_core function, approx line 4214. Still some effort required in backtracking from there to determine the reason for the dropped packets.

[network-shark]

I also see these these drops on my pi3

pi@raspberrypi:~ $ uname -a Linux raspberrypi 4.9.13-v7+ #974 SMP Wed Mar 1 20:09:48 GMT 2017 armv7l GNU/Linux

pi@raspberrypi:~ $ ifconfig eth0 Link encap:Ethernet HWaddr b8:27:eb:05:2e:7f inet addr:192.168.10.11 Bcast:192.168.10.255 Mask:255.255.255.0 inet6 addr: fe80::xxxxxxx/64 Scope:Link inet6 addr: 2003:xxxxxx/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14465964 errors:0 dropped:276326 overruns:0 frame:0

If you need more infos feel free to ask !

[stamster]

@JamesH65 so we can conclude issue is located in the kernel core? On some cloud providers I also experience dropped packets in RX line.