4.14.27-v7+ / 3+ VLAN hw csum failure

[sinistermidget]

Adding a VLAN to eth0 and then putting any traffic over it results in the following error regularly repeating:

[ 1349.736843] eth0.20: hw csum failure
[ 1349.736865] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C      4.14.27-v7+ #1100
[ 1349.736870] Hardware name: BCM2835
[ 1349.736904] [<8010fff8>] (unwind_backtrace) from [<8010c260>] (show_stack+0x20/0x24)
[ 1349.736922] [<8010c260>] (show_stack) from [<8076dd84>] (dump_stack+0xd4/0x118)
[ 1349.736943] [<8076dd84>] (dump_stack) from [<8067665c>] (netdev_rx_csum_fault+0x44/0x48)
[ 1349.736962] [<8067665c>] (netdev_rx_csum_fault) from [<80668f7c>] (__skb_checksum_complete+0xb4/0xb8)
[ 1349.736980] [<80668f7c>] (__skb_checksum_complete) from [<806f997c>] (icmp_rcv+0xd0/0x388)
[ 1349.736997] [<806f997c>] (icmp_rcv) from [<806bfaf4>] (ip_local_deliver_finish+0xe4/0x330)
[ 1349.737012] [<806bfaf4>] (ip_local_deliver_finish) from [<806c0360>] (ip_local_deliver+0x54/0xdc)
[ 1349.737026] [<806c0360>] (ip_local_deliver) from [<806bff7c>] (ip_rcv_finish+0x23c/0x494)
[ 1349.737038] [<806bff7c>] (ip_rcv_finish) from [<806c0704>] (ip_rcv+0x31c/0x554)
[ 1349.737053] [<806c0704>] (ip_rcv) from [<80673c24>] (__netif_receive_skb_core+0x340/0xc84)
[ 1349.737068] [<80673c24>] (__netif_receive_skb_core) from [<806767d0>] (__netif_receive_skb+0x20/0x7c)
[ 1349.737083] [<806767d0>] (__netif_receive_skb) from [<806768c4>] (process_backlog+0x98/0x148)
[ 1349.737100] [<806768c4>] (process_backlog) from [<8067aba0>] (net_rx_action+0x2e8/0x45c)
[ 1349.737116] [<8067aba0>] (net_rx_action) from [<80101694>] (__do_softirq+0x18c/0x3d8)
[ 1349.737132] [<80101694>] (__do_softirq) from [<801237b4>] (irq_exit+0xe0/0x144)
[ 1349.737149] [<801237b4>] (irq_exit) from [<801754d8>] (__handle_domain_irq+0x70/0xc4)
[ 1349.737165] [<801754d8>] (__handle_domain_irq) from [<80101504>] (bcm2836_arm_irqchip_handle_irq+0xa8/0xac)
[ 1349.737181] [<80101504>] (bcm2836_arm_irqchip_handle_irq) from [<807899bc>] (__irq_svc+0x5c/0x7c)
[ 1349.737188] Exception stack(0x80c01ef0 to 0x80c01f38)
[ 1349.737198] 1ee0:                                     00000000 0574e8c0 397c6000 00000000
[ 1349.737210] 1f00: 80c00000 80c03dcc 80c03d68 80c81ffe 00000001 80b5fa30 babffa40 80c01f4c
[ 1349.737221] 1f20: 80c04174 80c01f40 80108a6c 80108a70 60000013 ffffffff
[ 1349.737239] [<807899bc>] (__irq_svc) from [<80108a70>] (arch_cpu_idle+0x34/0x4c)
[ 1349.737255] [<80108a70>] (arch_cpu_idle) from [<80789114>] (default_idle_call+0x34/0x48)
[ 1349.737271] [<80789114>] (default_idle_call) from [<80161170>] (do_idle+0xd8/0x150)
[ 1349.737285] [<80161170>] (do_idle) from [<80161484>] (cpu_startup_entry+0x28/0x2c)
[ 1349.737299] [<80161484>] (cpu_startup_entry) from [<80782e64>] (rest_init+0xbc/0xc0)
[ 1349.737317] [<80782e64>] (rest_init) from [<80b00df8>] (start_kernel+0x3d4/0x3e0)

Using ethtool to turn off hw csum offload as a workaround stops the message from reappearing.

Furthermore, attempting to copy a large file over the VLAN interface causes scp to stall. Adjusting the interface's MTU from 1500 to 1496 as a workaround resolves that issue.

There are no issues using the same SD card on an original Pi 3B (no +).

The issue has been replicated on two separate 3B+ boards using multiple power supplies, including the newly sanctioned 2.5A model.

These results have been replicated with the following system configurations:

• 2018-03-13-raspbian-stretch-lite image with no modifications (kernel version 4.9.80-v7+)
• 2018-03-13-raspbian-stretch-lite after running rpi-update (kernel version 4.14.27-v7+)
• Gentoo with kernel version 4.9.80-v7
• Gentoo with kernel version 4.15.10-v7

[fuxjezz]

I'm using vlans as well on up2date raspbian and think I'm affected by the same issue. Additional sympoms include high cpu usage (because of interrupt storm?) and very bad network connection.
Putting the sdcards back into "regular" Pi 3B's = no trouble whatsoever with those same installations/sdcards).

Thank you for sharing your workarounds. I will try to implement them tomorrow on a couple of 3B+'s that are kind of useless because of this issue :(

If there are any tests or if there is any information I can contribute: please let me know.

Kind regards,

Ruben

[15610.881778] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W       4.14.29-v7+ #1101
[15610.881781] Hardware name: BCM2835
[15610.881797] [<8010fff8>] (unwind_backtrace) from [<8010c260>] (show_stack+0x20/0x24)
[15610.881810] [<8010c260>] (show_stack) from [<8076e024>] (dump_stack+0xd4/0x118)
[15610.881824] [<8076e024>] (dump_stack) from [<80676898>] (netdev_rx_csum_fault+0x44/0x48)
[15610.881839] [<80676898>] (netdev_rx_csum_fault) from [<806691b8>] (__skb_checksum_complete+0xb4/0xb8)
[15610.881853] [<806691b8>] (__skb_checksum_complete) from [<806e8680>] (tcp_v4_rcv+0x648/0xca4)
[15610.881866] [<806e8680>] (tcp_v4_rcv) from [<806bfd2c>] (ip_local_deliver_finish+0xe4/0x330)
[15610.881878] [<806bfd2c>] (ip_local_deliver_finish) from [<806c0598>] (ip_local_deliver+0x54/0xdc)
[15610.881890] [<806c0598>] (ip_local_deliver) from [<806c01b4>] (ip_rcv_finish+0x23c/0x494)
[15610.881901] [<806c01b4>] (ip_rcv_finish) from [<806c093c>] (ip_rcv+0x31c/0x554)
[15610.881913] [<806c093c>] (ip_rcv) from [<80673e60>] (__netif_receive_skb_core+0x340/0xc84)
[15610.881926] [<80673e60>] (__netif_receive_skb_core) from [<80676a0c>] (__netif_receive_skb+0x20/0x7c)
[15610.881941] [<80676a0c>] (__netif_receive_skb) from [<8067a6c8>] (netif_receive_skb_internal+0x30/0xe0)
[15610.881956] [<8067a6c8>] (netif_receive_skb_internal) from [<8067a79c>] (netif_receive_skb+0x24/0x98)
[15610.882015] [<8067a79c>] (netif_receive_skb) from [<7f87140c>] (br_netif_receive_skb+0x48/0x60 [bridge])
[15610.882117] [<7f87140c>] (br_netif_receive_skb [bridge]) from [<7f8714dc>] (br_pass_frame_up+0xb8/0x11c [bridge])
[15610.882219] [<7f8714dc>] (br_pass_frame_up [bridge]) from [<7f871694>] (br_handle_frame_finish+0x110/0x4e0 [bridge])
[15610.882320] [<7f871694>] (br_handle_frame_finish [bridge]) from [<7f871bdc>] (br_handle_frame+0x178/0x2d4 [bridge])
[15610.882379] [<7f871bdc>] (br_handle_frame [bridge]) from [<80673ef0>] (__netif_receive_skb_core+0x3d0/0xc84)
[15610.882393] [<80673ef0>] (__netif_receive_skb_core) from [<80676a0c>] (__netif_receive_skb+0x20/0x7c)
[15610.882407] [<80676a0c>] (__netif_receive_skb) from [<80676b00>] (process_backlog+0x98/0x148)
[15610.882421] [<80676b00>] (process_backlog) from [<8067addc>] (net_rx_action+0x2e8/0x45c)
[15610.882435] [<8067addc>] (net_rx_action) from [<80101694>] (__do_softirq+0x18c/0x3d8)
[15610.882446] [<80101694>] (__do_softirq) from [<801237b4>] (irq_exit+0xe0/0x144)
[15610.882460] [<801237b4>] (irq_exit) from [<80175554>] (__handle_domain_irq+0x70/0xc4)
[15610.882473] [<80175554>] (__handle_domain_irq) from [<80101504>] (bcm2836_arm_irqchip_handle_irq+0xa8/0xac)
[15610.882486] [<80101504>] (bcm2836_arm_irqchip_handle_irq) from [<80789c3c>] (__irq_svc+0x5c/0x7c)
[15610.882491] Exception stack(0x80c01ef0 to 0x80c01f38)
[15610.882499] 1ee0:                                     00000000 26bdd9d0 3c363000 00000000

top on rb3plus system:

top - 20:38:27 up  4:20,  1 user,  load average: 3.85, 1.89, 1.27
Tasks:  85 total,   3 running,  46 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :   994136 total,   768520 free,    40664 used,   184952 buff/cache
KiB Swap:  1106812 total,  1106812 free,        0 used.   836096 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    7 root      20   0       0      0      0 R  89.1  0.0  21:23.75 ksoftirqd/0

2796 packets transmitted, 260 packets received, 90.7% packet loss
round-trip min/avg/max/stddev = 1.425/333511.060/1076825.483/310975.815 ms

2743 packets transmitted, 773 packets received, +1 duplicates, 71.8% packet loss
round-trip min/avg/max/stddev = 0.268/418017.065/1241204.172/344407.543 ms

(ping stats from other systems on network)

[klpgo]

Same problem here. Thanks for sharing the workaround. But I cannot confirm the workaround. Setting MTU to 1496 on the vlan still produces hw csum error.
My configuration is: raspbian-stretch-lite image with no modifications (kernel version 4.9.80-v7+)

[6by9]

Sorry, I haven't got a VLAN environment set up at the moment (I'll try to remember next week), but in looking at the throughput issues I've stumbled across the setup for the watchdog to abort frames that are too long.
https://github.com/raspberrypi/linux/blob/rpi-4.14.y/drivers/net/usb/lan78xx.c#L2155
VLAN tagging will add 4 bytes to your packet, taking you right up to the abort watchdog time.
If you can rebuild your own kernel, then could you try increasing the value to

buf |= (((size + 12) << MAC_RX_MAX_SIZE_SHIFT_) & MAC_RX_MAX_SIZE_MASK_);

to give a small amount more headroom before frames get aborted. Decreasing the mtu likewise returns you to a default packet size and are therefore within the standard timeout.

[klpgo]

Upgraded my raspbian-stretch-lite image to kernel version 4.14.31-v7+. But that does not change the problem.

[6by9]

Upgraded my raspbian-stretch-lite image to kernel version 4.14.31-v7+. But that does not change the problem.

I wouldn't expect it to seeing as the issue is reported on 4.14.27 and there have been no significant changes between that and 4.14.31.

[vintozver]

same problem. works fine with raspberry pi 3 b but doesn't work with 3 b+

[6by9]

I've confirmed to my own satisfaction that the timeout I suspected was to blame.
Pull Request created, and then hopefully VLAN tagged data work as expected.

[popcornmix]

Latest rpi-update kernel has a potential fix for this issue. Please test.

[klpgo]

Thanks for the fix. I've upgraded to kernel 4.14.32-v7+ and tested in on a pi3 b+. A part of the problem seems to be fixed. The frequency of the error is very much reduced. But it still appears from time to time.

I found that I can trigger the error by pinging the vlan interface from the same vlan. I see no dropped packages/transmit errors on the interface.

This is the dmesg output on a not by ping triggered error:

[  279.552533] eth0.40: hw csum failure
[  279.552565] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C      4.14.32-v7+ #1106
[  279.552571] Hardware name: BCM2835
[  279.552619] [<8010fff8>] (unwind_backtrace) from [<8010c260>] (show_stack+0x20/0x24)
[  279.552645] [<8010c260>] (show_stack) from [<80783ca4>] (dump_stack+0xd4/0x118)
[  279.552678] [<80783ca4>] (dump_stack) from [<8068a898>] (netdev_rx_csum_fault+0x44/0x48)
[  279.552701] [<8068a898>] (netdev_rx_csum_fault) from [<8067d1b8>] (__skb_checksum_complete+0xb4/0xb8)
[  279.552720] [<8067d1b8>] (__skb_checksum_complete) from [<8070e2ec>] (icmp_rcv+0xd0/0x388)
[  279.552739] [<8070e2ec>] (icmp_rcv) from [<806d3e34>] (ip_local_deliver_finish+0xe4/0x330)
[  279.552758] [<806d3e34>] (ip_local_deliver_finish) from [<806d46ec>] (ip_local_deliver+0x54/0xdc)
[  279.552780] [<806d46ec>] (ip_local_deliver) from [<806d42fc>] (ip_rcv_finish+0x27c/0x4e0)
[  279.552800] [<806d42fc>] (ip_rcv_finish) from [<806d4a90>] (ip_rcv+0x31c/0x554)
[  279.552822] [<806d4a90>] (ip_rcv) from [<80687e60>] (__netif_receive_skb_core+0x340/0xc84)
[  279.552844] [<80687e60>] (__netif_receive_skb_core) from [<8068aa0c>] (__netif_receive_skb+0x20/0x7c)
[  279.552866] [<8068aa0c>] (__netif_receive_skb) from [<8068ab00>] (process_backlog+0x98/0x148)
[  279.552887] [<8068ab00>] (process_backlog) from [<8068eddc>] (net_rx_action+0x2e8/0x45c)
[  279.552905] [<8068eddc>] (net_rx_action) from [<80101694>] (__do_softirq+0x18c/0x3d8)
[  279.552922] [<80101694>] (__do_softirq) from [<801237b4>] (irq_exit+0xe0/0x144)
[  279.552941] [<801237b4>] (irq_exit) from [<80175554>] (__handle_domain_irq+0x70/0xc4)
[  279.552958] [<80175554>] (__handle_domain_irq) from [<80101504>] (bcm2836_arm_irqchip_handle_irq+0xa8/0xac)
[  279.552975] [<80101504>] (bcm2836_arm_irqchip_handle_irq) from [<8079f8bc>] (__irq_svc+0x5c/0x7c)
[  279.552990] Exception stack(0x80c01ef0 to 0x80c01f38)
[  279.553009] 1ee0:                                     00000000 01027c3c 397c4000 00000000
[  279.553027] 1f00: 80c00000 80c03dcc 80c03d68 80c88172 00000001 80b60a30 babffa40 80c01f4c
[  279.553041] 1f20: 80c04174 80c01f40 80108a6c 80108a70 60000013 ffffffff
[  279.553061] [<8079f8bc>] (__irq_svc) from [<80108a70>] (arch_cpu_idle+0x34/0x4c)
[  279.553078] [<80108a70>] (arch_cpu_idle) from [<8079f034>] (default_idle_call+0x34/0x48)
[  279.553095] [<8079f034>] (default_idle_call) from [<801611ec>] (do_idle+0xd8/0x150)
[  279.553109] [<801611ec>] (do_idle) from [<80161500>] (cpu_startup_entry+0x28/0x2c)
[  279.553130] [<80161500>] (cpu_startup_entry) from [<80798d84>] (rest_init+0xbc/0xc0)
[  279.553151] [<80798d84>] (rest_init) from [<80b00df8>] (start_kernel+0x3d4/0x3e0)

[6by9]

@kgottschalk Can you try sudo ethtool -K eth0 rx off to confirm that disabling hw rx checksumming solves the problem? I think you are seeing something unrelated, so need to diagnose that in isolation now that we have (hopefully) resolved the first issue.
Pings are normally never anywhere near the full MTU, so your issue won't be the rx jabber watchdog firing. Are you really doing just ping <Pi IP addr>, or is there something else involved?

@sinistermidget Could you retest under your original conditions please?