Workqueue: kmmcd mmc_rescan crash with Raspbian kernel 4.4.38-v7 #1885
Comments
I doubt this is a firmware issue. It could actually be a combo of a bad card plus a bug in the driver. It might well be that the driver thinks you've removed the card even though you haven't. And since pulling out the SD card while the RPI is running is a no-no, I could see why that might not have been tested well. I know some effort is being made to replace this driver with an upstream version. Also 4.9 is now considered the stable version of the kernel on the RPI. |
I have completely write tested the card with I just ordered another Pi 3 and a SanDisk Ultra 64GB SDXC (80 MB/Sek) Class 10 to rule out hardware issues. |
|
You missed of the header of that kernel, but a stall at that point looks like a symptom that something else hasn't released the MMC device, possibly because something failed during a previous operation. The sdhost driver will log error messages, so in the absence of those I'm leaning towards the ARM<->VPU mailbox stalling while trying to configure the clock for the SD interface. Errors like this are often the sign of an inadequate power supply. What does "vcgencmd get_throttled" report on your system after some typical activity?
No - it's still the next version until we make a Raspbian release using it. |
|
That is puzzling then... Personally, I would have tried a very, very cheap card first to rule out hardware issue. Like the kind you get for $5 in the US. I've actually collected a whole bunch of those over the years. What I meant by cheap cards, is the all these SD cards are cheap. I don't care how much you pay for them, they are essentially designed for the camera/camcorder/phone market so they don't typically get the abuse of something that's may well be on all the time. |
|
@pelwell I actually see mailbox stalls from time to time and I'm not 100% sure it's the power supply since I can sometimes see it even if the RPI is connected to a bench variable power supply that can do 5 amps. I've found that especially in 64 bit, the RPI is unstable at 1.2GHz. Setting the max at 1.0GHz seems to help. But I also haven't ruled out a firmware issue. |
|
@pelwell What kernel header to you need? I am using the default Raspbian 8 Jessie kernel My power supply seems stable and sufficient as concluded from By the way if you google for |
|
Meanwhile I noticed that this problem is triggered by the chromium-browser (Version 51.0.2704.91 Built on Ubuntu 14.04, running on Raspbian 8.0) when several sites are open. |
|
Please update to the latest kernel which may help, 4.9 is now the release version. Please report back the results. |
JamesH65
added
the
Waiting for external input
|
Unfortunately I still see the same problem after the upgrade. |
JamesH65
added
Bug
and removed
Waiting for external input
|
I never saw this before, and now saw it a bunch of times running 4.9.80-v7+ during a very fast wget download. I guess this is what made the Raspberry Pi 2 B temporarily unresponsive, but it didn't crash and the download didn't fail. |
|
Does this still happen on the latest 4.14 kernel? This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested. |
JamesH65
added
the
Close within 30 days
|
unfortunately it is: and my kernel is:
|
|
OK, we need to fathom out what is going on that might cause this. Since this appears to be very uncommon, is there anything about your setup that could be regarded as a little unusual that may promote it? What are you doing when it happens? |
JamesH65
removed
the
Close within 30 days
|
I also have the same problem, What happens is that because of this after a while the pi starts killing tasks, and the pi is in somewhat of a zombie state. This is a pi used for display in the cafeteria, so I usually just reboot it instead of trying to connect to it through SSH to see what's going on. Next time it crashes I'll try to see wether the OS is still responsive or if the pi is really in some sort of zombie state. The pi crashed around We can see it "works" even tho the display is frozen. I have found another interesting thing: The dates don't match, but maybe there's something to dig here more generally. What is weird is that I have another pi setup exactly the same way and this one does not crash. I'll try to check the version differences, I think the hw is a tiny bit different too. |
|
Same here, but it only happens since I upgraded from jessie to stretch: This happens regularly after some time during relatively heavy writing (tvheadend recording) to a nfs mounted folder [49641.471064] INFO: task kworker/0:2:15266 blocked for more than 120 seconds. |
|
'me too', and I can reproduce it on two different 3B+, with different SD cards, and different power supplies. The stock kernel in raspbian-stretch-lite 2018-06-27 does it, 4.14.70-v7+ #1144 does it, 4.14.73-v7+ #1148 does it. This is a plain install, with ssh enabled, and my workload is pretty repetitive: Try to wear out the SD card. The loop goes: Kit with instructions: https://cm.kasperkp.dk/githubbug-pi3bp-1.tgz [33002.940127] gogo (5419): drop_caches: 3 Occasionally the temporary hang is long enough that I lose ssh. |
|
I had the idea to test this on various models, to see if multicore or wifi was the trigger: No, it is Pi3/Pi3B+. Pi 0W (4pcs): 48 hours runtime, no failures Pi 2 original (1pc): 48 hours runtime, no failures Pi 3 original (1pc): 8 hours runtime, 6 failures, throttled=0x0 Pi 3B+ (3pcs): Highly variable. The best has 6 days of runtime and only a single failure, the worst fails once per hour on average. Various things tried: |
|
I see there are no updates in this issue for a while now... Also got this issue since I've upgraded to stretch I'm running multiple docker containers, one uses an NFS share might that be causing it? (transmission) I use this to mount it: Here are the log entries from syslog: |
|
Same for me. System got back operating after about 15 minutes of being unresponsive and spamming the kern.log file. 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux |
|
Since a couple of weeks I have the same error. But I have a raspberry pi 2. I've got a Samsung 32gb SD card in it. I'm running influxdb on the card, next to Hass, grafana, traefik and Telegraph, all in Docker containers. I monitor the stats of the raspberry pi, but see no problem (enough memory, no cpu spike). Maybe it's because too Mach io on SD card? After a reboot it's up and running fine, even for a week. |
|
I have the same problem since a few month ago, every node of my Raspberry Pi Kubernetes Cluster has every day (nearly every 24 hours) that exception. As nodes i use Raspberry Pi 3 B+ and Raspberry Pi 3. Kernel: Linux k8s-node3 4.19.42-v7+ #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l GNU/Linux
|
|
Can anyone who is affected by this issue, test this upstream patch also available in rpi-5.1.y ? |
|
I have ported the patch locally back to the 4.19.y branch and build the kernel to test the patch. System is rebooted and if the error happens again, it will be in the next 24-28 hours... |
|
@lategoodbye It seems that the patch is not working, in the morning i have the same errors as before.
|
|
@sbiermann The patch isn't intended for this issue. So it was only a try. It would be helpful for me, if you could reproduce this issue with the mmc/host/bcm2835 (upstream) instead of mmc/host/bcm2835-sdhost (downstream). Both use the same DT compatible so you will need to change your kernel config accordingly (enable MMC_BCM2835 and disable MMC_BCM2835_SDHOST). Thanks |
|
@lategoodbye Sorry, i'm not so familar with building a kernel, normally i use only default kernel images or in a very few cases rpi-update given kernel. This https://www.raspberrypi.org/documentation/linux/kernel/building.md instructions i have followed to build a 4.19.y kernel with your changes of the drivers/dma/bcm2835-dma.c. Now i have changed my .config file so i looks like:
Is that correct? |
|
Yes, but you also need to have (sorry about the confusing names)
|
|
That doesn't work and brokes the kernel, the Pi never boots up. To recover i have copied the old kernel back to the system and now it starts up again. |
|
I have solved the issue for me, but it is not a general solutions. Anyway i describe here my workarround maybe it helps some other to solve the problem. |
|
I still have this bug with 5.15.30-v8+ 64 bit Raspberry Pi 3B+.... the system freezed and resumed after 3 hours of freeze. |
|
I finding the same issue on newer kernel |
|
In the Linux kernel version 4.19.11, I encountered a similar issue. During debugging, it was observed that the This is just my hypothesis, and I have not verified it. |
|
Also running into this issue on a Raspberry Pi 5 with latest updates. Started after updating the system to 6.6.62 (sadly do not know what I had before). Disabled swap that this is not written to SD. Because I have 2 BTRFS drives attached, I decided to try the fix: We will see it it crashes within 24 hours, I always had this crash once a day in the morning. |
My Raspberry Pi 3 runs unstable and crashes every few days. Before the crash /var/log/messages shows the following error repeatedly with a separation of a few minutes:
I am running Raspbian 8.0 (Jessie) with kernel
Linux raspberrypi 4.4.38-v7+ #938 SMP Thu Dec 15 15:22:21 GMT 2016 armv7l GNU/Linux
I am also discussing this at
http://raspberrypi.stackexchange.com/q/61050/58211.
I have also reported this at raspberrypi/firmware#731 but I think it fits better here. Unfortunately I cannot move the issue, so its kind of a duplicate, now. Maybe it should be deleted at firmware.
The text was updated successfully, but these errors were encountered: