Kernel warning with 5.10.52-v8+ #1441 SMP PREEMPT Tue Aug 3 18:14:03 BST 2021 aarch64

[jfikar]

I'm getting a strange kernel warning after upgrade to the latest kernel via apt update in 64bit Raspberry Pi OS. This is on Raspberry Pi4 with 8GB and running from USB3 disk. I haven't seen this warning with previous kernel versions. Kernel version is
5.10.52-v8+ #1441 SMP PREEMPT Tue Aug 3 18:14:03 BST 2021 aarch64.

[ 4649.560224] ------------[ cut here ]------------
[ 4649.560256] WARNING: CPU: 2 PID: 1706 at kernel/rcu/tree_plugin.h:297 rcu_note_context_switch+0x64/0x428
[ 4649.560259] Modules linked in: z3fold zstd zstd_compress 8021q garp stp llc binfmt_misc sg brcmfmac brcmutil cfg80211 rfkill raspberrypi_hwmon bcm2835_codec(C) bcm2835_v4l2(C) bcm2835_isp(C) bcm2835_mmal_vchiq(C) v4l2_mem2mem videobuf
2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common vc_sm_cma(C) videodev mc rpivid_mem uio_pdrv_genirq nvmem_rmem uio i2c_dev aes_neon_bs sha256_generic aes_neon_blk crypto_simd cryptd ip_tables x_tables ipv6
[ 4649.560356] CPU: 2 PID: 1706 Comm: chia_full_node Tainted: G         C        5.10.52-v8+ #1441
[ 4649.560359] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[ 4649.560364] pstate: 20000085 (nzCv daIf -PAN -UAO -TCO BTYPE=--)
[ 4649.560368] pc : rcu_note_context_switch+0x64/0x428
[ 4649.560373] lr : rcu_note_context_switch+0x54/0x428
[ 4649.560375] sp : ffffffc013c5bac0
[ 4649.560378] x29: ffffffc013c5bac0 x28: ffffff8106875ac0 
[ 4649.560385] x27: 0000000000000000 x26: ffffffc011476000 
[ 4649.560391] x25: 0000000000000000 x24: ffffff81fefb9240 
[ 4649.560397] x23: 0000000000000000 x22: ffffff8106875ac0 
[ 4649.560403] x21: ffffffc011462328 x20: ffffffc011298948 
[ 4649.560409] x19: ffffff81fefb9ec0 x18: 0000000000000000 
[ 4649.560415] x17: 0000000000000000 x16: 0000000000000000 
[ 4649.560421] x15: 0000000000000000 x14: 0000000000000000 
[ 4649.560427] x13: 00000000312b1000 x12: 0000000000000000 
[ 4649.560433] x11: 0000000000000000 x10: ffffff81bbf31270 
[ 4649.560440] x9 : ffffffc010acffb8 x8 : ffffff808a0b7480 
[ 4649.560446] x7 : ffffff81016f8000 x6 : ffffffc013c5bab0 
[ 4649.560451] x5 : 0000000000000001 x4 : ffffffc1edfdd000 
[ 4649.560457] x3 : 0000000000000001 x2 : ffffffc010fc7000 
[ 4649.560463] x1 : ffffffc1edfdd000 x0 : 0000000000000001 
[ 4649.560470] Call trace:
[ 4649.560475]  rcu_note_context_switch+0x64/0x428
[ 4649.560480]  __schedule+0xc8/0x818
[ 4649.560484]  schedule+0x48/0x100
[ 4649.560487]  io_schedule+0x24/0xb0
[ 4649.560492]  __lock_page_or_retry+0x1cc/0x498
[ 4649.560497]  do_swap_page+0x1f4/0x700
[ 4649.560501]  handle_mm_fault+0x4a8/0xcb8
[ 4649.560505]  do_page_fault+0x148/0x3e8
[ 4649.560509]  do_translation_fault+0x58/0x70
[ 4649.560516]  do_mem_abort+0x48/0xa8
[ 4649.560520]  el0_da+0x40/0x58
[ 4649.560525]  el0_sync_handler+0x68/0xb8
[ 4649.560529]  el0_sync+0x180/0x1c0
[ 4649.560533] ---[ end trace 114c06ed393e9fe3 ]---
[ 5126.441263] TCP: eth0: Driver has suspect GRO implementation, TCP performance may be compromised.

[pelwell]

Errors like that are almost exclusively caused by undervoltage - a small change in instruction ordering due to a kernel version bump can be enough to make a theoretical problem real. I hope you are powering the USB drive separately.

[HiassofT]

This seems to be this stable regression https://lore.kernel.org/r/[email protected] which was fixed in 5.10.53 (commits https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=7d7d0e84ac0d80815abfb7db3ba3e683830b310d and https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=277b311ae170b90fcc30551d1a30d17fceb113a4)

Time to update the rpi-5.10.y branch :-)

[jfikar]

Thanks @HiassofT, that's useful.

@pelwell I don't think it is related to low voltage.

$ vcgencmd get_throttled
throttled=0x0

Currently SSD drive is connected directly and another HDD via powered USB hub. I can try to connect SSD also through the hub.

[jfikar]

So the same thing happens with all USB drives connected through powered USB hub. Only this time it happens at uptime 6628s, so after TCP: eth0: Driver has suspect GRO implementation, TCP performance may be compromised. But I don't think it means something.

[popcornmix]

There is a rpi-update available which may fix this issue. Please update and report.

[jfikar]

It looks good after upgrading and 9 hours uptime. Good job, thanks

[popcornmix]

Thanks for confirming.