4.14.74-v7+ 3b+ VLAN udp IPv4/6 - hw csum failure

[Wireheadbe]

Hi All - seems like issue #2458 is popping up again.

Setup is eth0 on local lan; eth0.30 with tagged data. Running IPv6 as well. (both over eth0.30 as tun0 for openvpn)
Having OpenVPN traffic comming in over eth0.30 cause lots of these errors:

[  862.694409] eth0: hw csum failure
[  862.694452] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C      4.14.74-v7+ #1149
[  862.694458] Hardware name: BCM2835
[  862.694495] [<8010ffd4>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
[  862.694514] [<8010c240>] (show_stack) from [<80788604>] (dump_stack+0xd4/0x118)
[  862.694537] [<80788604>] (dump_stack) from [<8068e870>] (netdev_rx_csum_fault+0x44/0x48)
[  862.694567] [<8068e870>] (netdev_rx_csum_fault) from [<80681144>] (__skb_checksum_complete+0xbc/0xc0)
[  862.694587] [<80681144>] (__skb_checksum_complete) from [<807331c8>] (nf_ip_checksum+0xd4/0x130)
[  862.694704] [<807331c8>] (nf_ip_checksum) from [<7f558a4c>] (udp_error+0x138/0x1c8 [nf_conntrack])
[  862.694894] [<7f558a4c>] (udp_error [nf_conntrack]) from [<7f551994>] (nf_conntrack_in+0xec/0x560 [nf_conntrack])
[  862.695004] [<7f551994>] (nf_conntrack_in [nf_conntrack]) from [<7f59f2dc>] (ipv4_conntrack_in+0x28/0x2c [nf_conntrack_ipv4])
[  862.695031] [<7f59f2dc>] (ipv4_conntrack_in [nf_conntrack_ipv4]) from [<806d04dc>] (nf_hook_slow+0x4c/0xd0)
[  862.695056] [<806d04dc>] (nf_hook_slow) from [<806d8df4>] (ip_rcv+0x460/0x514)
[  862.695075] [<806d8df4>] (ip_rcv) from [<8068be4c>] (__netif_receive_skb_core+0x340/0xc84)
[  862.695094] [<8068be4c>] (__netif_receive_skb_core) from [<8068e9e4>] (__netif_receive_skb+0x20/0x7c)
[  862.695112] [<8068e9e4>] (__netif_receive_skb) from [<8068ead8>] (process_backlog+0x98/0x148)
[  862.695128] [<8068ead8>] (process_backlog) from [<80692df0>] (net_rx_action+0x2e8/0x45c)
[  862.695144] [<80692df0>] (net_rx_action) from [<80101694>] (__do_softirq+0x18c/0x3d8)
[  862.695161] [<80101694>] (__do_softirq) from [<80123870>] (irq_exit+0x108/0x164)
[  862.695179] [<80123870>] (irq_exit) from [<801759a8>] (__handle_domain_irq+0x70/0xc4)
[  862.695195] [<801759a8>] (__handle_domain_irq) from [<80101504>] (bcm2836_arm_irqchip_handle_irq+0xa8/0xac)
[  862.695211] [<80101504>] (bcm2836_arm_irqchip_handle_irq) from [<807a41bc>] (__irq_svc+0x5c/0x7c)
[  862.695218] Exception stack(0x80c01ef0 to 0x80c01f38)
[  862.695228] 1ee0:                                     00000000 045b6fe8 397c2000 00000000
[  862.695241] 1f00: 80c00000 80c03dcc 80c03d68 80c885b2 00000001 80b60a30 babff9c0 80c01f4c
[  862.695252] 1f20: 80c04174 80c01f40 80108a4c 80108a50 60000013 ffffffff
[  862.695268] [<807a41bc>] (__irq_svc) from [<80108a50>] (arch_cpu_idle+0x34/0x4c)
[  862.695291] [<80108a50>] (arch_cpu_idle) from [<807a393c>] (default_idle_call+0x34/0x48)
[  862.695309] [<807a393c>] (default_idle_call) from [<801614b8>] (do_idle+0xd8/0x150)
[  862.695324] [<801614b8>] (do_idle) from [<801617cc>] (cpu_startup_entry+0x28/0x2c)
[  862.695339] [<801617cc>] (cpu_startup_entry) from [<8079d664>] (rest_init+0xbc/0xc0)
[  862.695359] [<8079d664>] (rest_init) from [<80b00df8>] (start_kernel+0x3d4/0x3e0)

Seems like issue 2458 is popping up again or wasn't fixed completely. Running

ethtool -K eth0 gro off 
ethtool -K eth0 rx-vlan-hw-parse off

Still results in these errors being generated.

[Wireheadbe]

#2659 seems similar - openvpn and iptables involved as well

[Wireheadbe]

Does look like it's coming in from ipv4 instead looking at the trace

[6by9]

@Wireheadbe
Do you have an easy way to reproduce?
I'm assuming that this has been running without logging errors in the past, or has this never worked cleanly for you?

Can you try reverting to an earlier kernel and retesting? sudo rpi-update <hash> where hash is the commit ID from https://github.com/Hexxeh/rpi-firmware/commits/master. Even a rough idea of when this has started failing would be useful.

We haven't made any changes since #2458 regarding the lan78xx driver, but it's possible something in the core has been backported to stable and is causing grief.
The fact that we're also seeing hw csum errors being reported without VLANs involved, and on SMSC95xx (#2712), imply it is something more generic and not #2458 having returned.

[Wireheadbe]

It's a pretty recent install. Only thing I added yesterday was DHCPv6 next to radvd. This morning I connected via OpenVPN and noticed them. That's the only change I had in terms of functionality. I was getting them on an earlier kernel; so i actually did an rpi-update. The issue remained. The offloading disable does make them happen less often.

[Wireheadbe]

went back to ec9d84e - seems to be gone at the moment? I'll keep an eye on it before confirming 100%

[Wireheadbe]

Did some speedtests / ipv4 / ipv6. VPN traffic etc etc.. still quiet at the moment on ec9d84e - 4.14.54-v7+ SMP Mon Jul 9 16:41:01 BST 2018 armv7l GNU/Linux

[6by9]

Thank you - that's a brilliant data point.

If you find that is stable, please can you try a couple of further bisections? Trying a5b781c (Sept 5) and then either e880e62 (Sept 18) if that is good, or 911147a (Aug 16) if not would be wonderful.
If you can narrow it down to a single commit then there's an even better chance of us finding the issue, but I do recognise that is quite an ask.
Currently we're still in the position of not being able to reproduce it, so we're totally stuck.

[Wireheadbe]

tried every single version from that - seems to happen from c919d63

[6by9]

Thank you. We can have a look at what changed with 4.14.71.

[6by9]

Hmm, 4.14.70 to 4.14.71 is 128 commits of which around 35 appear to be network related.

Can you give some more details of your setup? OpenVPN is running on the Pi, with the tunnel coming in on eth0.30? We need some way to reproduce this unless you fancy bisecting the kernel commits (not as bad as it sounds, but a tad involved).

[Wireheadbe]

I don't even have to run the tunnel. I'll see if i can get around to pinpointing it with wireshark. Bisecting the commits - please elaborate. Maybe this is the easiest way, although being somewhat repetitive ;)

[Wireheadbe]

I think I have it narrowed down to packets sent from one device - were there any changes in how IGMPv2; STP or ICMPv6 packets are handled? That's the big three red flags I have..

[6by9]

See https://www.raspberrypi.org/documentation/linux/kernel/building.md for details of how to compile your own kernel.

git is the source control system used for the kernel source. If includes a mechanism for bisecting the tree to close in on a regression as quickly as possible. See https://git-scm.com/docs/git-bisect for a description.
4.14.71 was commit 1244bbb, 4.14.70 was commit 1244bbb. Your comment that "seems to happen from c919d63" says that it should be one of the commits between the two that caused the issue. bisecting 128 commits should take 7 builds if I've got my numbers right.

[6by9]

You can see the commit history from that time for yourself via https://github.com/raspberrypi/linux/commits/rpi-4.14.y?before=1244bbb3e92135d247e2dddfa6fe5e3e171a9635+35

6bf32cd looks like it might have the potential for changing the behaviour on checksums. If you were to build your own kernel then it'd be worth a quick test having done git revert 6bf32cda46ebfbaf13da3c48a0a009adae925703