Dropped RX packets - RPi 1, 2, and 3 #1954
Comments
|
Any relevant messages in syslog? Also, any ideas when this problems started to occur? |
JamesH65
added
Waiting for external input
JamesH65
removed
the
Waiting for external input
|
A quick glance and some minor debugging at the driver level seems to indicate that it is not the driver itself that is dropping the packet - the code that increments the rx_dropped counter is not called. So presumably somewhere else in the stack is deciding to drop the packet. I'm not sure how to find out where in the stack though! Will investigate further. |
|
It's quite possible this is harmless. The dropped packet count is not only used for errors, but also flagging up packets that are dropped because the linux stack doesn't handle them (some IPv6 stuff if you don't have IPv6 enabled for example). This might be one of those messages. I will try and find out which it is. |
|
It's not IPv6 related as first thing I usually do after installation is
disabling IPv6 on all interfaces via sysctl.conf kernel parameter.
Is it harmless? You never really know, dropped packets shouldn't be there
for sure.
19.05.2017. 17.11, "James Hughes" <[email protected]> је написао/ла:
… It's quite possible this is harmless. The dropped packet count is not only
used for errors, but also flagging up packets that are dropped because the
linux stack doesn't handle them (some IPv6 stuff if you don't have IPv6
enabled for example). This might be one of those messages. I will try and
find out which it is.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1954 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMNhf-Eu7da9v0nABpFCHCfdfwgiJGI5ks5r7bEhgaJpZM4M33E8>
.
|
|
OK, not IPv6, but that was only given as an example of some packets that are simply discarded by the network stack. Dropped packets are not, necessarily, an error. If the stack gets a correctly formatted packet it doesn't implement, then it is dropped. It's not an error as such. So it is perfectly reasonable to be getting dropped packets and not worry about them. However, in this case I would like to know whether this is the case here. |
|
Here is some (probably quite old) text on why a packet may be dropped. Also worth noting that even if IPv6 is disable on the Pi, it could still receive IPv6 packets from other devices on the network, which will cause dropped packets
|
|
After much faffing, I have build a utility I found called dropwatch, rebuilt the kernel to turn on a particular form of net logging and now have a list of addresses where packets are being dropped. However, cross references those addresses to the kernel isn't giving valid results, so i suspect all the drops are in modules. What is worth noting is that the ifconfig dropped packets counter is a small subset of the actual number of packets dropped. I can see which address in my list is causing the ifconfigs drops, just need to figure out what code corresponds to that location. |
|
OK, thanks to a timely intervention from @pelwell I do have some idea of the code that is dropping the packets. The ifconfig dropped packet counter appears to be incremented in the __netif_receive_skb_core function, approx line 4214. Still some effort required in backtracking from there to determine the reason for the dropped packets. |
|
I also see these these drops on my pi3
If you need more infos feel free to ask ! |
|
@JamesH65 so we can conclude issue is located in the kernel core? On some cloud providers I also experience dropped packets in RX line. |
|
I don't think any conclusion can be made since I don't yet know the cause of the dropped packets - it could be entirely benign. I think everyone sees the dropped packets on the Pi, I certainly have been seeing them for years, but ignored then. |
|
Well, it started to happen only after kernel/firmware update year ago or so. On this version I have zero dropped packets: Heh, it might be that this old version has a bug of not incrementing counter 😆 |
|
Or newer version added more places where the counter could be incremented... |
|
stamster, can you identify the exact update which caused this. See: If you click on each commit the end of the url contains a git hash. Run (I suspect it will be one of the major bumps - e.g. the first 4.4 kernel) |
|
I've been super busy recently - I'll try to experiment with down-grades this weekend. |
|
I was doing some testing today with the latest Raspbian release, 5GB of data transferred with no dropped packets. This was on ethernet. If doing testing check the latest release first. |
|
Well, I got that kernel 5 days ago and still there were dropped packets. Let's see... |
|
Let me add my stats: RX drop was about 4% with unpatched rasbian-jessie. |
|
I still have drops, even with latest kernel.
|
|
Hmm, just been doing some testing on this, not seeing dropped packets on the ethernet at all. It is possible that this is environmental? |
|
A similar issue has just come up on the linux networking list, unrelated to Pi. Suggestion is RX queueing problem. I'm not sure how to go further here, since I cannot see any dropped packets on the ethernet connection at all. This is a continuation of the previous stats, after leaving it over the weekend, not though with anything hammering the connection. |
|
@stamster Could you give me some idea of how your network is set up? Since I (nd others) am seeing no errors on the onboard Ethernet at all, it would be interesting to know if you have some odd setup that might be causing an issue. |
JamesH65
added
the
Waiting for external input
|
ping @stamster Are you still seeing this? If so are there any strange things on your network that might have some effect. Also, I'm currently using Stretch with a 4.13 kernel and seeing no drops. It would be interesting, if you have the ability to build your own kernel and install it, to see if you still see errors with the latest kernel code. |
|
I tested with over 10GB of data sent back and forth with no dropped packets at all. This is on Stretch with a 4.13 kernel (admittedly not yet a release kernel, but will be released eventually). I'm inclined, unless I hear otherwise to close this issue. |
|
I have seen discussions elsewhere that seemed to indicate that the ethernet connection is the most power hungry, and low power or inconsistent power to the RPi results in RX errors as the most obvious symptom. You may want to verify that the correct power is being supplied. |
|
[ Unconstructive diversion deleted - I don't want to lock this thread ] |
|
You may as well lock it since you're not addressing the issue anyway. That's how things work around here... |
|
No, it really isn't how things work around here. If we close the issue, we no longer have it visible, so it gets forgotten and will never get fixed. Right now, it's visible, and when time allows someone will look at it. However, we are a tiny team, with a lot to do, and since this only reduces bandwidth, rather than killing anything completely, it's lower priority than many other issues. |
|
Not having a reliable way to reproduce an issue also makes it incredibly hard to work on. Reports of it being seen on a Pi4 make it even stranger as Pi3B+ and Pi4 have a totally different ethernet interfaces to Pi1/2/3B, which makes it seem more systemic than just the ethernet chip driver. Pi4 is over a totally different interface too (inbuilt as opposed to USB). |
|
@6by9 |
|
I have this issue with RX dropped packets on all of my Ubuntu 20.04 LTS Server on Raspberry Pi 4 8GB model as you can see below: Are there any workarounds or fixes planned? |
|
Adding a bit of info to this.
|
|
This will never be fixed or acknowledged. The broadcom chips they use for
the Pis are garbage with garbage IP. They can't ban you from github but
they will ban you from their forums if you point out the flaws and make too
many questions.
And yes, that's how things work around here (see my post on this thread
from 2021, if there's any still not deleted).
Alvaro S.
…On Wed, Sep 25, 2024 at 1:11 AM Bruno Franca dos Reis < ***@***.***> wrote:
Adding a bit of info to this.
1. The issue happens on both my Pi 4b and my Pi 5 (the 4b I bought a
few years back, the 5 is brand new).
2. It happens whether they are connected via my wi-fi or wired to my
switch.
3. It happens in any networks I put them.
4. Other devices in my network don't show this behavior.
5. I'm getting roughly 1 RX packet drop every 5-10 seconds on the Pi
4, *regardless of volume of packets or data*. This is incredibly
consistent — I enabled monitoring via Grafana and I can see it's incredibly
stable at a minute average of 133m packet/s dropped (i.e. 0.133 ~= 1/7.5).
6. One apparent correlation is that, if I have an SSH connection
established to the Pi, it will systematically drop within tens of seconds
to 1 or 2 minutes, with a "connection corrupted" error message, and showing
an error of a packet with unreasonably large length; these issues seem to
happen at exactly the same instant when there's an RX packet dropped,
although I couldn't scientifically validate this. SSH is so unusable that I
have to use mosh instead. One thread I found online suggested changing
ciphers, but it happens regardless of the cipher used.
7. I don't see any system logs that correlate with either the SSH
connection drops, or the RX packet drops.
—
Reply to this email directly, view it on GitHub
<#1954 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AF3ARF5P2HYD5FAD3H5CNADZYHWSBAVCNFSM6AAAAABOZJUH46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZSGU2DONRXG4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
This is very fishy - 4B and 5 have nothing in common when it comes to Ethernet except the PHY. And this thread is about Pis 1-3, which are different again - their Ethernet was attached via USB, and the OTG USB host controller has well-known limitations. Pi 4 uses the on-board GENET controller, while Pi 5 uses the MAC on RP1.
I've never seen any kind of unreliability on Ethernet, other than a few issues with Energy Efficient Ethernet when the link is being brought up.
There are no secret problems we can't/don't talk about - there are some which are widely experienced and easily reproduced, and others which seem to only affect a handful of people and probably the result of unknown environmental factors.
Comments which are abusive have been, and will continue to be, deleted. Feel free to ask awkward questions, particularly if they include new information - "me too" posts don't help anyone. |
|
Hi @pelwell - thanks for the incredibly quick reply, I really appreciate it!
Indeed - I didn't know these details, but assumed they were very different. I was actually very surprised to see the same issue on the 5!
Yup, it's all very strange! By reading through the thread, it seems that you folks are having a hard time being able to repro the issue, and I'd love to provide more data in case that could help. Do you have any suggestions of what else I could try to diagnose? Do you think some screen recording from my SSH client could be useful? Let me know if you can think of any diagnostic steps beyond what I shared above. FWIW, these Pi 4b and 5 aren't really in any kind of "production" situation, so I'm happy to try things out and provide data, even if they're on the riskier side. Thanks again! |
|
i. What is at the other end of your Ethernet cable(s)? |
|
Many years ago I had a problem with ethernet dropping out when I used the Pi camera. It was a power supply issue. When the camera kicked in, the voltage dropped enough for the ethernet to drop out, but everything else continued normally. Could power be an issue? |
My network is eero-based. A gateway eero connects to my ISP and to a switch. The switch then goes to 2 more eeros, and and to a few other devices (Xbox, AVR, a caldigit ts3 where I dock my laptop). And there's a bunch of devices on the wifi. Pretty standard home network. Specifically regarding the Pi(S): on Ethernet, I tried connecting them to the switch, as well as directly to the eeros, as well as wifi. In all cases I'm seeing the same RX drop issue.
Yup, different cables, different ports of the eero, different eeros, etc... 😔
No errors, just drops (ie |
Interesting! Just had one thought now that you mention. For context - The Pi 5 is the one from CanaKit with the M.2 HAT+ and NVMe. It also has an active fan. Nothing else AFAIK. I've tried both the 45W PD adapter that came with it, as well as a 140W MacBook charger. The Pi 4b is simpler, just the Pi, a MicroSD (tried both an older 8GB slower one, as well as a 128GB newer and faster), and a fan (connect to GND and +5v). I tried with a 5.1V 3A adapter, as well as the 140W MacBook charger. The one interesting thing I noticed on the 4b is that if I connect vs disconnect the fan the behavior does change a bit. I'll grab some data on this. Also, I haven't done any such experiments with the 5. I'll share the data tomorrow. Thanks for the ideas, folks! |
|
From what I can see reading the code, dropped packets have been abandoned by the network stack, either after they have been received or before they are transmitted, because it can't cope with them at that time - possibly as a result of memory exhaustion. |
|
Thinking about memory exhaustion: |
|
vii. Is it possible that something on the network is generating "interesting" broadcast traffic every second? |
|
A few more things I noticed:
Answering your questions @pelwell:
2GB total, barely any used:
This is the contents currently: https://gist.github.com/bfreis/82b19ad44e0e6d08d74a969ce870a52f
I put here some info, including the output of systemctl | grep running, and raspinfo: https://gist.github.com/bfreis/8f66dd1a45892c18e454c52f59a07620
Maybe the eeros? I'm not sure. I'll investigate and report back. The 5s and 10s extremely regular cadence seems really suspicious. |
|
Ok, I think we're getting closer now! Some new info:
So I think we can assume that all those RX dropped are caused by those 0x9104 broadcast from the eeros! I wonder if the other folks reporting this similar issue above also had something broadcasting weird packets in their network. I'm gonna see what I can correlate with the SSH issue, and share any findings. Thanks for the ideas! |
|
Ok, I figured it out! I used My investigation then led me to the issue: my Macbook! It seems that the connection was being closed on my end, rather than on the Pi's end. Apparently there's some sort of bug on the Firewall stack on MacOS Sequoia that randomly kills SSH and SSL connections. And it gets even more strange: it seems to only affect my connections when using iTerm2, but it works fine when using Apple's Terminal app! It's impacting Chrome, etc. Another fun fact: if I try to download a large file via HTTPS with |
|
Congratulations - you did most of the heavy lifting. |
Hello,
I have multiple RPi's running on two locations. I noticed that each of them report dropped packets, but only in RX direction.
RPi 3:
RPi 2:
Just rebooted after latest rpi-firmware patch, and only 4 mins of uptime getting:
RX packets:689 errors:0 dropped:5 overruns:0 frame:0RPi 1 model B:
RX packets:647 errors:0 dropped:28 overruns:0 frame:0RPi 1:
RX packets:209844 errors:0 dropped:14892 overruns:0 frame:0The only single RPi which does not seem to be affected by this issue is another RPi 1 model B, where I didn't upgraded firmware.
So if it's a firmware related issue, the last known good firmware/kernel w/o RX drops is this one:
10 days uptime, and not a single drop:
RPi's behave exactly the same on other networks, so it's not LAN related, since other PC's and devices do not have any drops with uptime of 250 days.
The text was updated successfully, but these errors were encountered: