Kernel Panic on Linux 4.9.35-v7+ #1014 SMP

[isgallagher]

[55604.367553] Unable to handle kernel NULL pointer dereference at virtual address 00000072
[55604.369603] pgd = b14e0000
[55604.370641] [00000072] *pgd=314be835, *pte=00000000, *ppte=00000000
[55604.371750] Internal error: Oops: 17 [#1] SMP ARM
[55604.372829] Modules linked in: overlay bnep hci_uart btbcm bluetooth ax88179_178a brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 spidev snd_pcm snd_timer snd i2c_bcm2835 spi_bcm2835 bcm2835_gpiomem uio_pdrv_genirq uio fixed i2c_dev ipv6
[55604.376541] CPU: 0 PID: 928 Comm: python3 Not tainted 4.9.35-v7+ #1014
[55604.377828] Hardware name: BCM2835
[55604.379097] task: b4e72c40 task.stack: b2c78000
[55604.380389] PC is at tcp_v4_early_demux+0x18/0x144
[55604.381685] LR is at tcp_v4_send_check+0x28/0x2c
[55604.382950] pc : [<8068fd5c>] lr : [<8068fca0>] psr: 60000013
sp : b2c79c90 ip : b2c79cb8 fp : b2c79cb4
[55604.385494] r10: 02080020 r9 : 0000aaf9 r8 : 0000ab00
[55604.386762] r7 : 0100007f r6 : 0100007f r5 : a5151a38 r4 : b2d61110
[55604.388049] r3 : a5151a38 r2 : 00000110 r1 : 0100007f r0 : 00000006
[55604.389340] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[55604.390678] Control: 10c5383d Table: 314e006a DAC: 00000055
[55604.392036] Process python3 (pid: 928, stack limit = 0xb2c78210)
[55604.393372] Stack: (0xb2c79c90 to 0xb2c7a000)
[55604.394664] 9c80: b2c79cb4 806127a8 00000000 a5151a38
[55604.397279] 9ca0: b2d61110 a5151a38 b2c79cd4 b2c79cb8 8068fca0 8068fd50 b2e99f40 a5151a38
[55604.399928] 9cc0: 00000020 b2d61110 b2c79ce4 b2c79cd8 8068fca0 8068fbd0 b2c79d2c b2c79ce8
[55604.402582] 9ce0: 80688180 8068fc84 00000002 00000000 00000000 0054661a 0054661a 00000000
[55604.405264] 9d00: 80261088 a5151980 b2e99f40 00005580 00000000 00005580 00000031 00000000
[55604.408080] 9d20: b2c79d94 b2c79d30 80688aa4 80687e6c 806117c0 00000800 8067a204 ffffffff
[55604.411057] 9d40: b2c79d74 b2c79d50 806116f8 000346db 80c05dcc 80c02d00 00000001 00000002
[55604.414158] 9d60: 00000005 00000000 b2c79db4 b2e99f40 a5151980 00005580 00000000 b2e99f40
[55604.417409] 9d80: 00000031 00000031 b2c79db4 b2c79d98 806895fc 8068878c 02080020 80461dd0
[55604.420730] 9da0: b2e99f40 b2e99f40 b2c79ddc b2c79db8 80677184 806895c4 00005580 00000031
[55604.424136] 9dc0: 00000031 00005580 c64ed2d6 a5151980 b2c79e74 b2c79de0 8067b388 80677068
[55604.427634] 9de0: 00005580 b2c79df0 80677ec4 8060eba8 00000000 80c06984 80d04818 80c0495c
[55604.431169] 9e00: b2c78020 00000000 00000001 00000000 00010000 00000000 00000001 b2c79ec8
[55604.434709] 9e20: b2e9a050 00000000 00000000 00005580 b1405015 00005580 7fffffff 00000000
[55604.438243] 9e40: 00000000 00000000 b2c78000 b2e99f40 00000000 b41248c0 00000121 80108244
[55604.441782] 9e60: b2c78000 00000000 b2c79e94 b2c79e78 806a513c 8067a978 8028fa98 8028fa04
[55604.445530] 9e80: 00000000 00000000 b2c79ea4 b2c79e98 8060a878 806a50ac b2c79f8c b2c79ea8
[55604.449362] 9ea0: 8060b914 8060a860 b2c79ec8 00000000 fffffff7 00000000 76669a08 00000031
[55604.452915] 9ec0: 00000000 00000000 00000001 00000000 00000000 b2c79ec0 00000000 00000000
[55604.456457] 9ee0: 00000000 00000000 00000000 00000000 5a678f09 3215e3db 5a678f09 3215e3db
[55604.459996] 9f00: 5a678f09 32ae7a1e 00000831 00000000 76e42138 7edf68f0 00000001 76e42138
[55604.463535] 9f20: 000000c3 80108244 b2c78000 00000000 b2c79fa4 b2c79f40 80275f30 802759fc
[55604.467075] 9f40: 00000831 00000000 0b300002 b2c781a4 00000001 00000000 00000000 00000000
[55604.470612] 9f60: 00000147 00000000 5a678f09 ffffffff 76a99b28 00000031 00000000 76a99b28
[55604.474146] 9f80: b2c79fa4 b2c79f90 8060b964 8060b858 00000000 00000000 00000000 b2c79fa8
[55604.477682] 9fa0: 801080c0 8060b948 00000031 00000000 00000004 76669a08 00000031 00000000
[55604.481217] 9fc0: 00000031 00000000 76a99b28 00000121 003c3848 01886be0 003c3848 7671acf0
[55604.484751] 9fe0: 00000000 7edf661c 001ed550 76f822cc 60000010 00000004 73736572 6372735f
[55604.488303] [<8068fd5c>] (tcp_v4_early_demux) from [<8068fca0>] (tcp_v4_send_check+0x28/0x2c)
[55604.491851] Code: e24cb004 e24dd010 e52de004 e8bd4000 (e5d0306c)
[55604.499475] ---[ end trace 83cda9367ff8f3fe ]---
[56202.780113] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
[56202.791225] pgd = b1524000
[56202.797242] [0000000c] *pgd=31421835, *pte=00000000, *ppte=00000000
[56202.803979] Internal error: Oops: 17 [#2] SMP ARM
[56202.810512] Modules linked in: overlay bnep hci_uart btbcm bluetooth ax88179_178a brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 spidev snd_pcm snd_timer snd i2c_bcm2835 spi_bcm2835 bcm2835_gpiomem uio_pdrv_genirq uio fixed i2c_dev ipv6
[56202.829045] CPU: 1 PID: 921 Comm: python3 Tainted: G D 4.9.35-v7+ #1014
[56202.840949] Hardware name: BCM2835
[56202.846555] task: b4d18ec0 task.stack: b14f0000
[56202.852046] PC is at tcp_recvmsg+0x544/0xa48
[56202.857488] LR is at __local_bh_enable_ip+0x9c/0x118
[56202.862655] pc : [<80678074>] lr : [<80122a8c>] psr: 20000013
sp : b14f1df0 ip : 80c040a4 fp : b14f1e5c
[56202.871789] r10: 00000000 r9 : 00000000 r8 : b520a96c
[56202.876169] r7 : b520a580 r6 : b520a604 r5 : b41ad500 r4 : 00000000
[56202.880381] r3 : 7fffffff r2 : 00000000 r1 : 80c05d48 r0 : b520aa40
[56202.884467] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[56202.888443] Control: 10c5383d Table: 3152406a DAC: 00000055
[56202.892504] Process python3 (pid: 921, stack limit = 0xb14f0210)
[56202.896620] Stack: (0xb14f1df0 to 0xb14f2000)
[56202.900544] 1de0: 80154d00 80c06984 80d04818 80c0495c
[56202.908049] 1e00: b14f0020 b14f1df0 00000000 b14f1eb8 80c05d48 00000000 00000000 b520aa40
[56202.915904] 1e20: 00000001 00010000 00000000 7fffffff 00f94359 b14f1eb8 b41ad500 00000000
[56202.924056] 1e40: 00000000 00000000 b14f0000 00000000 b14f1e8c b14f1e60 806a4a80 80677b3c
[56202.933166] 1e60: 00000000 b14f1e6c b14f1e84 00000000 8028fa98 00000000 b41ad500 00000000
[56202.942714] 1e80: b14f1e9c b14f1e90 80609228 806a49e8 b14f1f8c b14f1ea0 8060ba14 8060920c
[56202.952699] 1ea0: b14f1ec0 8060a860 fffffff7 00000000 01ebb6f8 00010000 00000000 00000000
[56202.963373] 1ec0: 00000000 00000000 00010000 b14f1eb0 00000001 b14f1ec0 00000000 00000000
[56202.974795] 1ee0: 00000000 00000000 00000000 8094e524 80c78fd0 003c388c b14f1f1c b14f1f08
[56202.987111] 1f00: 807193a4 801e35a0 00000000 00000000 b14f1f44 b14f1f20 80148814 80122c44
[56202.999304] 1f20: 80b7e940 80c0442c b14f1f44 b14f1f38 80122c44 801801e4 b14f1f64 b14f1f48
[56203.012105] 1f40: 80149be0 80122ba8 00000002 80b7cec0 00000000 00000001 b14f1f94 b14f1f68
[56203.025176] 1f60: 8010e0e4 80149b6c 00000004 ffffffff 767754f8 00000000 00000123 80108244
[56203.038962] 1f80: b14f1fa4 b14f1f90 8060ba98 8060b978 00000000 00000000 00000000 b14f1fa8
[56203.052989] 1fa0: 801080c0 8060ba7c ffffffff 767754f8 00000004 01ebb6f8 00010000 00000000
[56203.067228] 1fc0: ffffffff 767754f8 00000000 00000123 76fe14d0 00000000 01d80ba8 00010000
[56203.082250] 1fe0: 00000000 7eb140f4 00214cc0 76f9109c 60000010 00000004 71755555 15d95755
[56203.096461] [<80678074>] (tcp_recvmsg) from [<806a4a80>] (inet_recvmsg+0xa4/0xb8)
[56203.110341] [<806a4a80>] (inet_recvmsg) from [<80609228>] (sock_recvmsg+0x28/0x2c)
[56203.124145] [<80609228>] (sock_recvmsg) from [<8060ba14>] (SyS_recvfrom+0xa8/0x104)
[56203.137205] [<8060ba14>] (SyS_recvfrom) from [<8060ba98>] (SyS_recv+0x28/0x30)
[56203.143882] [<8060ba98>] (SyS_recv) from [<801080c0>] (ret_fast_syscall+0x0/0x1c)
[56203.156141] Code: e51b3030 e3530000 0a00006f e51b2048 (e592200c)
[56203.162660] ---[ end trace 83cda9367ff8f3ff ]---
HM3_dmesg.txt

[pelwell]

As bug reports go this is one of the more useless - there's no indication of what you were doing at the time, whether it is reproducible, whether you have made any configuration changes, etc.

Random errors in well used bits of code like the IP stack are usually caused by power supply problems, especially on a Pi3 with external hardware attached and drawing power from the Pi.

[isgallagher]

What we were doing at the time: Polling for data on the SPI bus via spidev python module, updating an on-system database, sending data out of the on-board Ethernet and a USB-to-Ethernet adapters, writing to disk (sd card) regularly, other calculations inside python scripts...

Is this reproducible? It happens very frequently. I'm not sure what exactly is causing it.

Configuration changes included changing hostname, changing /etc/network/interfaces file, changing locale settings to US, enabling SPI, serial, I2c, camera, ssh, changing memory split to 16.

We do have a USB to Ethernet adapter plugged in, and we are feeding data both through that and the on-board Ethernet adapter.

The power supply we are using is 5.0 V +/- 0.1 V, capable of supplying up to over 3 amps.

[JamesH65]

By removing things from the 'activities' list, at which point does the fault go away? Or do you have a much more minimal test case that exhibits the issue? There are so many things mentioned going on, that it will be borderline impossible to determine what the issue is without reducing the complexity.

[isgallagher]

I forgot to mention we have data on wlan0 sporadically. I am in the process of reducing the complexity to hopefully pinpoint what the trigger is by removing the USB to Ethernet port and disabling wlan0.

Thanks for your time.

[JamesH65]

@isgallagher Any progress to report on this?

[JamesH65]

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

[JamesH65]

Closing due to lack of activity. Please request to be reopened if you feel this issue is still relevant.

[isgallagher]

@JamesH65 Not sure why or which kernel version this was fixed in, but since we moved to 4.14.15-v7+ we have not seen this issue again.