Crash loop after kernel upgrade to 5.10.52

[tstevens]

Describe the bug
After an apt-get kernel update, Pi no longer boots and enters a crash loop when powered on. Kernel was upgraded from 5.10.17 to 5.10.52. Once started the Pi reaches the same step in the kernel log before hanging and restarting again. Watchdog timer is enabled which may be causing the restarts after each hang.

To reproduce
Power on Raspberry PI. Do not know if this happens after every upgrade to 5.10.52, this is the only Pi I have.

Expected behaviour
Successful boot.

Actual behaviour
Crash loop of Pi, failing to start. After reaching [ OK ] Found device /dev/disk/by-partuuid/18138b91-01. logs stop and nothing else is output until Pi resets.

System

Model: Raspberry Pi 2 B v1.1
Kernel version: 5.10.52
OS: Raspbian
Firmware: Pulled from Serial logs.
"2021-08-03T18:17:07, variant start"
"Firmware hash is 40787ee5905644f639a2a0f6e00ae12e517a2211"

Logs
Kernel logs from direct serial connection

[    0.000000] Booting Linux on physical CPU 0xf00
[    0.000000] Linux version 5.10.52-v7+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1441 SMP Tue Aug 3 18:10:09 BST 2021
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Raspberry Pi 2 Model B Rev 1.1
[    0.000000] random: fast init done
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] Reserved memory: created CMA memory pool at 0x16c00000, size 384 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003bffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000003bffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000003bffffff]
[    0.000000] percpu: Embedded 20 pages/cpu s50700 r8192 d23028 u81920
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 243600
[    0.000000] Kernel command line: coherent_pool=1M snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=Composite-1:720x480@60i,margin_left=32,margin_right=32,margin_top=32,margin_bottom=32 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=PARTUUID=18138b91-02 rootfstype=ext4 elevator=deadline fsck.repair=yes cgroup_enable=cpuset cgroup_enable=memory swapaccount=1 rootwait
[    0.000000] Kernel parameter elevator= does not have any effect anymore.
[    0.000000] Please use sysfs to set IO scheduler for individual devices.
[    0.000000] cgroup: Enabling cpuset control group subsystem
[    0.000000] cgroup: Enabling memory control group subsystem
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 562668K/983040K available (10240K kernel code, 1312K rwdata, 2948K rodata, 1024K init, 863K bss, 27156K reserved, 393216K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 32043 entries in 63 pages
[    0.000000] ftrace: allocated 63 pages with 6 groups
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] random: get_random_bytes called from start_kernel+0x3ac/0x580 with crng_init=1
[    0.000000] arch_timer: cp15 timer(s) running at 19.20MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[    0.000006] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[    0.000020] Switching to timer-based delay loop, resolution 52ns
[    0.000247] Console: colour dummy device 80x30
[    0.000894] printk: console [tty1] enabled
[    0.000954] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000)
[    0.000998] pid_max: default: 32768 minimum: 301
[    0.001175] LSM: Security Framework initializing
[    0.001384] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.001426] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.002566] CPU: Testing write buffer coherency: ok
[    0.003078] CPU0: update cpu_capacity 1024
[    0.003111] CPU0: thread -1, cpu 0, socket 15, mpidr 80000f00
[    0.004242] Setting up static identity map for 0x100000 - 0x10003c
[    0.004445] rcu: Hierarchical SRCU implementation.
[    0.005269] smp: Bringing up secondary CPUs ...
[    0.006355] CPU1: update cpu_capacity 1024
[    0.006365] CPU1: thread -1, cpu 1, socket 15, mpidr 80000f01
[    0.007542] CPU2: update cpu_capacity 1024
[    0.007550] CPU2: thread -1, cpu 2, socket 15, mpidr 80000f02
[    0.008669] CPU3: update cpu_capacity 1024
[    0.008677] CPU3: thread -1, cpu 3, socket 15, mpidr 80000f03
[    0.008821] smp: Brought up 1 node, 4 CPUs
[    0.008932] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[    0.008955] CPU: All CPU(s) started in HYP mode.
[    0.008974] CPU: Virtualization extensions available.
[    0.009870] devtmpfs: initialized
[    0.023076] VFP support v0.3: implementor 41 architecture 2 part 30 variant 7 rev 5
[    0.023356] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.023407] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.036851] pinctrl core: initialized pinctrl subsystem
[    0.037990] NET: Registered protocol family 16
[    0.041269] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[    0.045754] audit: initializing netlink subsys (disabled)
[    0.046120] audit: type=2000 audit(0.040:1): state=initialized audit_enabled=0 res=1
[    0.046680] thermal_sys: Registered thermal governor 'step_wise'
[    0.047478] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[    0.047530] hw-breakpoint: maximum watchpoint size is 8 bytes.
[    0.047812] Serial: AMBA PL011 UART driver
[    0.064617] bcm2835-mbox 3f00b880.mailbox: mailbox enabled
[    0.080180] raspberrypi-firmware soc:firmware: Attached to firmware from 2021-08-03T18:17:07, variant start
[    0.090186] raspberrypi-firmware soc:firmware: Firmware hash is 40787ee5905644f639a2a0f6e00ae12e517a2211
[    0.127105] Kprobes globally optimized
[    0.132104] bcm2835-dma 3f007000.dma: DMA legacy API manager, dmachans=0x1
[    0.133978] SCSI subsystem initialized
[    0.134233] usbcore: registered new interface driver usbfs
[    0.134305] usbcore: registered new interface driver hub
[    0.134380] usbcore: registered new device driver usb
[    0.136131] clocksource: Switched to clocksource arch_sys_counter
[    1.183340] VFS: Disk quotas dquot_6.6.0
[    1.183476] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    1.183667] FS-Cache: Loaded
[    1.183962] CacheFiles: Loaded
[    1.184654] simple-framebuffer 3eade000.framebuffer: framebuffer at 0x3eade000, 0x10a800 bytes, mapped to 0x(ptrval)
[    1.184698] simple-framebuffer 3eade000.framebuffer: format=a8r8g8b8, mode=656x416x32, linelength=2624
[    1.188303] Console: switching to colour frame buffer device 82x26
[    1.193193] simple-framebuffer 3eade000.framebuffer: fb0: simplefb registered!
[    1.204596] NET: Registered protocol family 2
[    1.206735] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    1.211707] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    1.215644] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    1.219605] TCP bind hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    1.221822] TCP: Hash tables configured (established 8192 bind 8192)
[    1.224141] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.226405] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.229188] NET: Registered protocol family 1
[    1.232091] RPC: Registered named UNIX socket transport module.
[    1.234253] RPC: Registered udp transport module.
[    1.236449] RPC: Registered tcp transport module.
[    1.238514] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.241990] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 5 counters available
[    1.249190] Initialise system trusted keyrings
[    1.251487] workingset: timestamp_bits=14 max_order=18 bucket_order=4
[    1.260872] zbud: loaded
[    1.264576] FS-Cache: Netfs 'nfs' registered for caching
[    1.267441] NFS: Registering the id_resolver key type
[    1.269384] Key type id_resolver registered
[    1.271214] Key type id_legacy registered
[    1.273097] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.274875] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    1.279342] Key type asymmetric registered
[    1.281040] Asymmetric key parser 'x509' registered
[    1.282743] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    1.286031] io scheduler mq-deadline registered
[    1.287775] io scheduler kyber registered
[    1.296225] bcm2835-rng 3f104000.rng: hwrng registered
[    1.298396] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
[    1.302722] gpiomem-bcm2835 3f200000.gpiomem: Initialised: Registers at 0x3f200000
[    1.317490] brd: module loaded
[    1.329997] loop: module loaded
[    1.333417] Loading iSCSI transport class v2.0-870.
[    1.336582] libphy: Fixed MDIO Bus: probed
[    1.338562] usbcore: registered new interface driver lan78xx
[    1.340346] usbcore: registered new interface driver smsc95xx
[    1.342078] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[    2.071887] Core Release: 2.80a
[    2.073558] Setting default values for core params
[    2.075199] Finished setting default values for core params
[    2.277220] Using Buffer DMA mode
[    2.278859] Periodic Transfer Interrupt Enhancement - disabled
[    2.280505] Multiprocessor Interrupt Enhancement - disabled
[    2.282118] OTG VER PARAM: 0, OTG VER FLAG: 0
[    2.283731] Dedicated Tx FIFOs mode
[    2.286360] 
[    2.286381] WARN::dwc_otg_hcd_init:1074: FIQ DMA bounce buffers: virt = 96d04000 dma = 0xd6d04000 len=9024
[    2.291100] FIQ FSM acceleration enabled for :
[    2.291100] Non-periodic Split Transactions
[    2.291100] Periodic Split Transactions
[    2.291100] High-Speed Isochronous Endpoints
[    2.291100] Interrupt/Control Split Transaction hack enabled
[    2.298558] 
[    2.298566] WARN::hcd_init_fiq:457: FIQ on core 1
[    2.301496] 
[    2.301503] WARN::hcd_init_fiq:458: FIQ ASM at 807c8de0 length 36
[    2.304517] 
[    2.304524] WARN::hcd_init_fiq:497: MPHI regs_base at bc810000
[    2.307583] dwc_otg 3f980000.usb: DWC OTG Controller
[    2.309167] dwc_otg 3f980000.usb: new USB bus registered, assigned bus number 1
[    2.310795] dwc_otg 3f980000.usb: irq 89, io mem 0x00000000
[    2.312385] Init: Port Power? op_state=1
[    2.313903] Init: Power Port (0)
[    2.315692] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[    2.318841] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.320518] usb usb1: Product: DWC OTG Controller
[    2.322122] usb usb1: Manufacturer: Linux 5.10.52-v7+ dwc_otg_hcd
[    2.323750] usb usb1: SerialNumber: 3f980000.usb
[    2.326062] hub 1-0:1.0: USB hub found
[    2.327803] hub 1-0:1.0: 1 port detected
[    2.330841] usbcore: registered new interface driver usb-storage
[    2.332862] mousedev: PS/2 mouse device common for all mice
[    2.335606] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[    2.339368] sdhci: Secure Digital Host Controller Interface driver
[    2.341105] sdhci: Copyright(c) Pierre Ossman
[    2.343364] sdhost-bcm2835 3f202000.mmc: could not get clk, deferring probe
[    2.345353] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.348049] ledtrig-cpu: registered to indicate activity on CPUs
[    2.350049] hid: raw HID events driver (C) Jiri Kosina
[    2.351914] usbcore: registered new interface driver usbhid
[    2.353652] usbhid: USB HID core driver
[    2.359164] Initializing XFRM netlink socket
[    2.360968] NET: Registered protocol family 17
[    2.362776] Key type dns_resolver registered
[    2.364870] Registering SWP/SWPB emulation handler
[    2.366704] registered taskstats version 1
[    2.368271] Loading compiled-in X.509 certificates
[    2.370591] Key type ._fscrypt registered
[    2.372137] Key type .fscrypt registered
[    2.373641] Key type fscrypt-provisioning registered
[    2.384497] uart-pl011 3f201000.serial: cts_event_workaround enabled
[    2.386306] 3f201000.serial: ttyAMA0 at MMIO 0x3f201000 (irq = 114, base_baud = 0) is a PL011 rev2
[    3.490130] printk: console [ttyAMA0] enabled
[    3.498017] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[    3.508918] sdhost: log_buf @ (ptrval) (d6d03000)
[    3.515511] Indeed it is in host mode hprt0 = 00021501
[    3.628742] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[    3.658524] of_cfs_init
[    3.662702] of_cfs_init: OK
[    3.668325] Waiting for root device PARTUUID=18138b91-02...
[    3.708675] mmc0: host does not support reading read-only switch, assuming write-enable
[    3.722243] mmc0: new high speed SDHC card at address 0001
[    3.730393] mmcblk0: mmc0:0001 EB1QT 29.8 GiB
[    3.738941]  mmcblk0: p1 p2
[    3.746226] usb 1-1: new high-speed USB device number 2 using dwc_otg
[    3.754586] Indeed it is in host mode hprt0 = 00001101
[    3.777107] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[    3.786304] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[    3.798691] EXT4-fs (mmcblk0p2): recovery complete
[    3.806749] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[    3.818398] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[    3.828230] devtmpfs: mounted
[    3.838030] Freeing unused kernel memory: 1024K
[    3.856430] Run /sbin/init as init process
[    3.996501] usb 1-1: New USB device found, idVendor=0424, idProduct=9514, bcdDevice= 2.00
[    4.008315] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.018220] hub 1-1:1.0: USB hub found
[    4.023996] hub 1-1:1.0: 5 ports detected
[    4.346197] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    4.476491] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00, bcdDevice= 2.00
[    4.488592] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.500697] smsc95xx v2.0.0
[    4.528601] systemd[1]: System time before build time, advancing clock.
[    4.613129] libphy: smsc95xx-mdiobus: probed
[    4.620342] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-3f980000.usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:ab:48:01
[    4.695813] NET: Registered protocol family 10
[    4.703746] Segment Routing with IPv6
[    4.779467] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[    4.811218] systemd[1]: Detected architecture arm.
[    4.905442] systemd[1]: Set hostname to <raspberrypi>.
[    4.915781] systemd[1]: Hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0
[    4.927767] systemd[1]: Set hardware watchdog to 10s.
[    5.134653] uart-pl011 3f201000.serial: no DMA platform data
[    5.685435] systemd[1]: /lib/systemd/system/fail2ban.service:12: PIDFile= references path below legacy directory /var/run/, updating /var/run/fail2ban/fail2ban.pid → /run/fail2ban/fail2ban.pid; please update the unit file accordingly.
[    5.816003] random: systemd: uninitialized urandom read (16 bytes read)
[    5.853046] random: systemd: uninitialized urandom read (16 bytes read)
[    5.862556] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    5.878777] random: systemd: uninitialized urandom read (16 bytes read)
[    5.888355] systemd[1]: Listening on Journal Socket.
[    5.904705] systemd[1]: Created slice system-getty.slice.
[    5.920790] systemd[1]: Mounting RPC Pipe File System...
[    6.178503] i2c /dev entries driver
[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Started Forward Password Re…ts to Plymouth Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Found device /dev/ttyAMA0.
[  OK  ] Found device /dev/disk/by-partuuid/18138b91-01.

Additional context
N/A

[popcornmix]

Can you post your config.txt and cmdline.txt?
Is hdmi connected? Any other peripherals connected?

[tstevens]

No HDMI connected nor peripherals.

cmdline.txt

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=18138b91-02 rootfstype=ext4 elevator=deadline fsck.repair=yes cgroup_enable=cpuset cgroup_enable=memory swapaccount=1 rootwait

config.txt

# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
#disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
arm_freq=1000

# Uncomment some or all of these to enable the optional hardware interfaces
dtparam=i2c_arm=off
#dtparam=i2s=on
dtparam=spi=off

# Uncomment this to enable the lirc-rpi module
#dtoverlay=lirc-rpi

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
#dtoverlay=vc4-fkms-v3d
max_framebuffers=2

[all]
#dtoverlay=vc4-fkms-v3d
#gpu_mem=160
gpu_mem=64
core_freq=500
sdram_freq=500
over_voltage=2
initial_turbo=60
max_usb_current=1
dtoverlay=vc4-kms-v3d,cma-384
dtoverlay=rpivid-v4l2
# Disable the ACT LED.
dtparam=act_led_trigger=none
dtparam=act_led_activelow=off
# Disable the PWR LED.
dtparam=pwr_led_trigger=none
dtparam=pwr_led_activelow=off
#dtoverlay=cma,cma-256
dtparam=audio=no
#enable_uart=1
#start_x=0
hdmi_blanking=2
hdmi_ignore_composite=1
hdmi_ignore_hotplug=1

[popcornmix]

Duplicate of #4457

[popcornmix]

(workaround is to remove dtoverlay=vc4-kms-v3d,cma-384 or connect hdmi until issue is resolved).