Kernel issue

[clemtfvr]

Hi!
I'm running a docker container for automated web tests (starts and stop chromium browser in the container every ~30secs) and the pi will hang after 1 to 2h running.
Ram is used 75% and CPU 25%, stable.

Kernel 4.14.79-v7+. Pi model B. Latest Raspbian stretch lite.
Internet connection is done through a Huawei E3372 4G Usb dongle. The pi is also connected to Ethernet.

I tried fresh install on 2 brand new sd and a usb SSD.
Changed power supply and cables to several high quality ones.

Typical dmesg logs (booting on SSD in this case):

[ 5036.076085] INFO: task jbd2/sda2-8:70 blocked for more than 120 seconds.
[ 5036.076123]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.076128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.076137] jbd2/sda2-8     D    0    70      2 0x00000000
[ 5036.076184] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.076206] [<8079f5d8>] (schedule) from [<8037b058>] (jbd2_journal_commit_transaction+0x1fc/0x17f0)
[ 5036.076227] [<8037b058>] (jbd2_journal_commit_transaction) from [<8038175c>] (kjournald2+0x110/0x2c0)
[ 5036.076246] [<8038175c>] (kjournald2) from [<8013dd98>] (kthread+0x13c/0x16c)
[ 5036.076265] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
[ 5036.076758] INFO: task kworker/u8:0:21638 blocked for more than 120 seconds.
[ 5036.076765]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.076772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.076778] kworker/u8:0    D    0 21638      2 0x00000000
[ 5036.076802] Workqueue: writeback wb_workfn (flush-8:0)
[ 5036.076853] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.076871] [<8079f5d8>] (schedule) from [<803772a0>] (wait_transaction_locked+0xb0/0xd4)
[ 5036.076891] [<803772a0>] (wait_transaction_locked) from [<803777e0>] (start_this_handle+0x474/0x744)
[ 5036.076909] [<803777e0>] (start_this_handle) from [<803780d0>] (jbd2__journal_start+0x104/0x210)
[ 5036.076928] [<803780d0>] (jbd2__journal_start) from [<8031c374>] (__ext4_journal_start_sb+0x6c/0xf8)
[ 5036.076946] [<8031c374>] (__ext4_journal_start_sb) from [<8033b648>] (ext4_writepages+0x54c/0x878)
[ 5036.076964] [<8033b648>] (ext4_writepages) from [<8022f798>] (do_writepages+0x30/0x8c)
[ 5036.076982] [<8022f798>] (do_writepages) from [<802bc1e4>] (__writeback_single_inode+0x44/0x430)
[ 5036.076999] [<802bc1e4>] (__writeback_single_inode) from [<802bcad8>] (writeback_sb_inodes+0x20c/0x4c4)
[ 5036.077015] [<802bcad8>] (writeback_sb_inodes) from [<802bce20>] (__writeback_inodes_wb+0x90/0xd0)
[ 5036.077032] [<802bce20>] (__writeback_inodes_wb) from [<802bd0a4>] (wb_writeback+0x244/0x358)
[ 5036.077048] [<802bd0a4>] (wb_writeback) from [<802bdaa4>] (wb_workfn+0x1d4/0x4c4)
[ 5036.077064] [<802bdaa4>] (wb_workfn) from [<801379b4>] (process_one_work+0x158/0x454)
[ 5036.077080] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
[ 5036.077094] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
[ 5036.077112] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
[ 5036.077160] INFO: task chromium-browse:28430 blocked for more than 120 seconds.
[ 5036.077167]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.077172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.077177] chromium-browse D    0 28430  28429 0x00000000
[ 5036.077202] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.077220] [<8079f5d8>] (schedule) from [<807a26d8>] (rwsem_down_read_failed+0x10c/0x15c)
[ 5036.077238] [<807a26d8>] (rwsem_down_read_failed) from [<807a1d6c>] (down_read+0x5c/0x60)
[ 5036.077255] [<807a1d6c>] (down_read) from [<807a5418>] (do_page_fault+0xfc/0x3a4)
[ 5036.077270] [<807a5418>] (do_page_fault) from [<801011e0>] (do_DataAbort+0x48/0xc4)
[ 5036.077286] [<801011e0>] (do_DataAbort) from [<807a4e64>] (__dabt_usr+0x44/0x60)
[ 5036.077295] Exception stack(0xa08a5fb0 to 0xa08a5ff8)
[ 5036.077305] 5fa0:                                     085c9ff0 00000000 00000021 0001eff1
[ 5036.077318] 5fc0: 00000021 085ca010 00000000 75c41818 75c417e4 085c9ff8 00000001 00000020
[ 5036.077329] 5fe0: 061899c8 7ebcff10 080e7008 75ba128e 200f0030 ffffffff
[ 5036.077338] INFO: task sandbox_ipc_thr:28436 blocked for more than 120 seconds.
[ 5036.077345]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.077350] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.077355] sandbox_ipc_thr D    0 28436  28429 0x00000000
[ 5036.077379] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.077397] [<8079f5d8>] (schedule) from [<807a26d8>] (rwsem_down_read_failed+0x10c/0x15c)
[ 5036.077414] [<807a26d8>] (rwsem_down_read_failed) from [<807a1d6c>] (down_read+0x5c/0x60)
[ 5036.077430] [<807a1d6c>] (down_read) from [<807a5418>] (do_page_fault+0xfc/0x3a4)
[ 5036.077443] [<807a5418>] (do_page_fault) from [<801011e0>] (do_DataAbort+0x48/0xc4)
[ 5036.077456] [<801011e0>] (do_DataAbort) from [<807a4af4>] (__dabt_svc+0x54/0x80)
[ 5036.077463] Exception stack(0x9208bb68 to 0x9208bbb0)
[ 5036.077473] bb60:                   9208bbf4 74205b64 00000000 00000000 00000000 74205b6a
[ 5036.077485] bb80: 00000002 00000000 00000001 00000001 b782b000 9208bf74 00000000 9208bbb8
[ 5036.077494] bba0: 9208bbec 802a09b4 200f0013 ffffffff
[ 5036.077510] [<807a4af4>] (__dabt_svc) from [<802a09b4>] (do_sys_poll+0x468/0x500)
[ 5036.077528] [<802a09b4>] (do_sys_poll) from [<802a1458>] (SyS_poll+0x94/0x11c)
[ 5036.077544] [<802a1458>] (SyS_poll) from [<80108000>] (ret_fast_syscall+0x0/0x28)
[ 5036.077554] INFO: task TaskSchedulerFo:28444 blocked for more than 120 seconds.
[ 5036.077561]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.077565] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.077570] TaskSchedulerFo D    0 28444  28429 0x00000000
[ 5036.077593] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.077611] [<8079f5d8>] (schedule) from [<807a26d8>] (rwsem_down_read_failed+0x10c/0x15c)
[ 5036.077629] [<807a26d8>] (rwsem_down_read_failed) from [<807a1d6c>] (down_read+0x5c/0x60)
[ 5036.077645] [<807a1d6c>] (down_read) from [<807a5418>] (do_page_fault+0xfc/0x3a4)
[ 5036.077658] [<807a5418>] (do_page_fault) from [<801011e0>] (do_DataAbort+0x48/0xc4)
[ 5036.077671] [<801011e0>] (do_DataAbort) from [<807a4e64>] (__dabt_usr+0x44/0x60)
[ 5036.077678] Exception stack(0xa0a0ffb0 to 0xa0a0fff8)
[ 5036.077687] ffa0:                                     00000000 00000080 00000000 7084c990
[ 5036.077700] ffc0: ffffff92 00000000 7084cab8 7084ca90 7084c9fc 7084c998 7084caa0 00000000
[ 5036.077711] ffe0: 000000f0 7084c950 76bab49f 76bad9f8 40000030 ffffffff
[ 5036.077720] INFO: task TaskSchedulerFo:28445 blocked for more than 120 seconds.
[ 5036.077726]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.077731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.077736] TaskSchedulerFo D    0 28445  28429 0x00000000
[ 5036.077760] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.077777] [<8079f5d8>] (schedule) from [<807a26d8>] (rwsem_down_read_failed+0x10c/0x15c)
[ 5036.077795] [<807a26d8>] (rwsem_down_read_failed) from [<807a1d6c>] (down_read+0x5c/0x60)
[ 5036.077815] [<807a1d6c>] (down_read) from [<804a2b18>] (__copy_to_user_memcpy+0xcc/0x190)
[ 5036.077833] [<804a2b18>] (__copy_to_user_memcpy) from [<804a2e74>] (arm_copy_to_user+0x20/0x2c)
[ 5036.077849] [<804a2e74>] (arm_copy_to_user) from [<802907fc>] (cp_new_stat64+0x17c/0x18c)
[ 5036.077866] [<802907fc>] (cp_new_stat64) from [<80290e34>] (SyS_fstat64+0x40/0x48)
[ 5036.077883] [<80290e34>] (SyS_fstat64) from [<80108000>] (ret_fast_syscall+0x0/0x28)
[ 5036.077891] INFO: task TaskSchedulerFo:28446 blocked for more than 120 seconds.
[ 5036.077898]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.077903] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.077908] TaskSchedulerFo D    0 28446  28429 0x00000000
[ 5036.077931] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.077948] [<8079f5d8>] (schedule) from [<807a26d8>] (rwsem_down_read_failed+0x10c/0x15c)
[ 5036.077966] [<807a26d8>] (rwsem_down_read_failed) from [<807a1d6c>] (down_read+0x5c/0x60)
[ 5036.077981] [<807a1d6c>] (down_read) from [<807a5418>] (do_page_fault+0xfc/0x3a4)
[ 5036.077996] [<807a5418>] (do_page_fault) from [<801011e0>] (do_DataAbort+0x48/0xc4)
[ 5036.078009] [<801011e0>] (do_DataAbort) from [<807a4e64>] (__dabt_usr+0x44/0x60)
[ 5036.078016] Exception stack(0xaf02ffb0 to 0xaf02fff8)
[ 5036.078026] ffa0:                                     00000000 00000080 00000000 6f6fe990
[ 5036.078038] ffc0: ffffff92 00000000 6f6feab8 6f6fea90 6f6fe9fc 6f6fe998 6f6feaa0 00000000
[ 5036.078049] ffe0: 000000f0 6f6fe950 76bab49f 76bad9f8 400d0030 ffffffff
[ 5036.078061] INFO: task TaskSchedulerFo:28455 blocked for more than 120 seconds.
[ 5036.078067]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.078072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.078077] TaskSchedulerFo D    0 28455  28429 0x00000000
[ 5036.078102] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.078117] [<8079f5d8>] (schedule) from [<80380e04>] (jbd2_log_wait_commit+0xac/0x134)
[ 5036.078132] [<80380e04>] (jbd2_log_wait_commit) from [<80382500>] (__jbd2_journal_force_commit+0x98/0x10c)
[ 5036.078148] [<80382500>] (__jbd2_journal_force_commit) from [<803825d0>] (jbd2_journal_force_commit+0x34/0x40)
[ 5036.078162] [<803825d0>] (jbd2_journal_force_commit) from [<8037077c>] (ext4_force_commit+0x34/0x40)
[ 5036.078180] [<8037077c>] (ext4_force_commit) from [<8032a9a8>] (ext4_sync_file+0x164/0x42c)
[ 5036.078200] [<8032a9a8>] (ext4_sync_file) from [<802c1450>] (vfs_fsync_range+0x68/0xc8)
[ 5036.078254] [<802c1450>] (vfs_fsync_range) from [<7f2938cc>] (ovl_dir_fsync+0x7c/0x104 [overlay])
[ 5036.078314] [<7f2938cc>] (ovl_dir_fsync [overlay]) from [<802c1450>] (vfs_fsync_range+0x68/0xc8)
[ 5036.078331] [<802c1450>] (vfs_fsync_range) from [<802c1538>] (do_fsync+0x4c/0x74)
[ 5036.078348] [<802c1538>] (do_fsync) from [<802c1830>] (SyS_fdatasync+0x1c/0x20)
[ 5036.078365] [<802c1830>] (SyS_fdatasync) from [<80108000>] (ret_fast_syscall+0x0/0x28)
[ 5036.078374] INFO: task TaskSchedulerFo:28456 blocked for more than 120 seconds.
[ 5036.078381]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.078386] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.078391] TaskSchedulerFo D    0 28456  28429 0x00000000
[ 5036.078416] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.078432] [<8079f5d8>] (schedule) from [<803772a0>] (wait_transaction_locked+0xb0/0xd4)
[ 5036.078449] [<803772a0>] (wait_transaction_locked) from [<803777e0>] (start_this_handle+0x474/0x744)
[ 5036.078465] [<803777e0>] (start_this_handle) from [<803780d0>] (jbd2__journal_start+0x104/0x210)
[ 5036.078481] [<803780d0>] (jbd2__journal_start) from [<8031c374>] (__ext4_journal_start_sb+0x6c/0xf8)
[ 5036.078497] [<8031c374>] (__ext4_journal_start_sb) from [<8033f7fc>] (ext4_dirty_inode+0x40/0x70)
[ 5036.078513] [<8033f7fc>] (ext4_dirty_inode) from [<802bbdec>] (__mark_inode_dirty+0x54/0x408)
[ 5036.078533] [<802bbdec>] (__mark_inode_dirty) from [<802a7dec>] (generic_update_time+0x84/0xc0)
[ 5036.078551] [<802a7dec>] (generic_update_time) from [<802a8134>] (file_update_time+0xc8/0x11c)
[ 5036.078571] [<802a8134>] (file_update_time) from [<802217d8>] (__generic_file_write_iter+0xa0/0x1f8)
[ 5036.078589] [<802217d8>] (__generic_file_write_iter) from [<80328d40>] (ext4_file_write_iter+0x188/0x454)
[ 5036.078610] [<80328d40>] (ext4_file_write_iter) from [<8028b7ec>] (__vfs_write+0xe0/0x138)
[ 5036.078627] [<8028b7ec>] (__vfs_write) from [<8028ba0c>] (vfs_write+0xb4/0x1bc)
[ 5036.078644] [<8028ba0c>] (vfs_write) from [<8028bde0>] (SyS_pwrite64+0x88/0xa0)
[ 5036.078660] [<8028bde0>] (SyS_pwrite64) from [<80108000>] (ret_fast_syscall+0x0/0x28)
[ 5036.078669] INFO: task TaskSchedulerFo:28457 blocked for more than 120 seconds.
[ 5036.078676]       Tainted: G         C      4.14.79-v7+ #1159
[ 5036.078680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5036.078685] TaskSchedulerFo D    0 28457  28429 0x00000000
[ 5036.078710] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
[ 5036.078728] [<8079f5d8>] (schedule) from [<807a26d8>] (rwsem_down_read_failed+0x10c/0x15c)
[ 5036.078746] [<807a26d8>] (rwsem_down_read_failed) from [<807a1d6c>] (down_read+0x5c/0x60)
[ 5036.078761] [<807a1d6c>] (down_read) from [<807a5418>] (do_page_fault+0xfc/0x3a4)
[ 5036.078775] [<807a5418>] (do_page_fault) from [<801011e0>] (do_DataAbort+0x48/0xc4)
[ 5036.078788] [<801011e0>] (do_DataAbort) from [<807a4e64>] (__dabt_usr+0x44/0x60)
[ 5036.078795] Exception stack(0x93795fb0 to 0x93795ff8)
[ 5036.078804] 5fa0:                                     00000000 00000080 00000000 66e6e990
[ 5036.078817] 5fc0: ffffff92 00000000 66e6eab8 66e6ea90 66e6e9fc 66e6e998 66e6eaa0 00000000
[ 5036.078827] 5fe0: 000000f0 66e6e950 76bab49f 76bad9f8 40000030 ffffffff

I can also provide with the python error resulting in the container but I'm not sure it can be helpful.
Best regards!
Clement

[clemtfvr]

I found what was going wrong. Chromium generates multiple files at startup and doesn't clean then properly when closed creating huge amount of leftover files. It's actually related to this issue: SeleniumHQ/docker-selenium#329
I'm using a dirty workaround for now, restarting the container every hour to wipe all these files.

This issue is therefore not directly related to kernel so I'm closing it.