Video decoder crash every few minutes #1627
Comments
|
This may be related to raspberrypi/linux#4592 Please check if "vcdbg log msg" contains messages with "recycling bust". |
|
Thanks for the tip, will give it a go! I've actually spent the last 12 hours replacing the Pi4 with an old Intel NUC because the problem was getting so bad. Not only was the video freezing but the Pi would get into a reboot loop and it would only display maybe 10 seconds of video before rebooting. After five or six reboots it would freeze at the rainbow screen, and since the watchdog can't be activated during boot, it was frozen until it was power cycled. I'm currently investigating watchdog alternatives as it seems it is very easy for the Pi to freeze during boot - I've lost count of the number of times I have had to power cycle it over the last few days. I tried to compare the UART output from a frozen and successful boot but there wasn't anything glaringly obvious, although sometimes it would randomly request a different file from the TFTP server which is odd. I never thought to try |
|
Ah. In my tests with 100+ Pi 4B, I've noticed that you absolutely want a case with sufficient thermal capacity or active cooling. Try a metal case with pillars on the hot chips. The Pi will slow down heavily if it gets too hot. Try to stay below 80°C if you want any reasonable performance. For my other use case (video capture + encoding) the Pi in the default plastic case did reach >80°C within 5-10 minutes and then the encoded frame rate went from 25 fps to 0.5 fps. A good metal case lowered the peak temperature to 65°C even in a 30°C room. Decoding is generating less heat than encoding, but the advice about overheating applies in either case. You can monitor temperature by running |
|
That's very interesting! I hadn't noticed much of a slowdown with the decoding (no visible framerate drop) but it was definitely throttling as the temperature was reported at 85°C which I believe is the max. Possibly it was hitting this so quickly (also 30°C ambient) that I was only ever seeing the throttled framerate. I did buy a PoE hat for it since it had a fan on it but never got a chance to install it. I read somewhere online someone from the Pi team saying it should be impossible to damage the Pi from overheating as it will just throttle itself, so I presumed the video freezing and reboots were firmware/driver bugs rather than thermal issues, however now I'm beginning to wonder. It's pretty good you have over 100 devices to test with, that gives a lot of valuable information. What are you using them for, out of curiosity? I'm using ffplay for video and Chromium in kiosk mode for displaying web page content, with command-line parameters to position the windows so I don't need to run a window manager. |
|
The 50x slowdown under throttling doesn't make a lot of sense. The ARM clock speed goes down from 1500 to 700, which is only a x2 change, so not sure where the rest of the speed drop is coming from. Odd. As for @Malvineous issues, to me, that sounds like a Pi that is failing/defective. Getting that hot that quickly sounds odd, although 30 ambient is pretty warm. I would not expect Pi freezes under throttling conditions, it's just slows down. And since most people are not seeing the level of video lock up etc I doubt its bugs in the decoder stack. Worth trying a fan or heat shedding case. Fan more likely to help with ambient that high. |
|
@JamesH65 The 50x slowdown in my setup may be caused by GStreamer dropping buffers which took too long to encode. @Malvineous Each device in my fleet hosts a GStreamer-based open source bidirectional (either transmit or receive, not both at the same time) video live streaming solution (including image/text overlays for transmission) for club lectures, churches, anyone with low budget. Centrally managed (for stream start/stop) with local interaction (e.g. speaker name as text overlay, slide display, other useful stuff). Right now only churches have adopted that stuff, but a lot of them. The whole thing is nonprofit because I believe that nonprofits should not pay through the nose for remote participation when they're trying to use physical/social distancing. I plan to submit a talk about it to the next FOSDEM conference. |
|
I ordered some metal cases with "pillars" pressing down on the chips and thermal pads to fit between the Pi chips and the pillars. The first one (a box shape with grooves in the top) lowered the temperature from 85°C down to a maximum of 53°C and completely stopped the freezing problem. The second case had a fan as well and dropped the temperature by a further 10°C to 43°C and likewise the Pi has been running for over 24 hours now without a single unexpected reboot. So this means the problem is definitely due to overheating, and apparently the throttling not being able to throttle back enough. I know the ARM core gets throttled when the temperature is too high, but does the VideoCore get throttled? In my case the ARM is sitting at around 22% CPU (~50% CPU on two cores, 0% on the other two cores) however I am using V4L2M2M to hardware decode two H264 streams. It would seem the H264 decoding is what is contributing most of the heat, and possibly that isn't getting throttled which is why it kept overheating? |
|
@Malvineous have you tried with Note: the h264 hardware creates negligible heat when compared to the arm. |
|
Sorry no I never got that far. I took the top of the heatsink case off to try it - 32°C ambient today so good day to test. I left it as-is to confirm it would reboot without any heatsink which it did, once it got to around 80°C. I then switched it off, set I then removed I had a pedestal fan in the room for the first test, and this accidentally provided enough of a breeze to keep the temperature at around 72°C and it was behaving normally there too. It wasn't until I blocked off enough airflow for it to get above 80°C that it rebooted. So looks like it's down to the throttling not being aggressive enough (or this unit being sensitive above 80°C rather than 85°C) and it's just overheating. |
|
Going to close this one now as it definitely looks like it was due to overheating. I suppose you could argue that it's still a bug in that the thermal throttling isn't fully working, but then you can also make the case that running it right up past the 80 degree mark is not very wise either. |
There seems to be some bugs in the H264 decoder. I am streaming video from another Raspberry Pi, and it was working fine, until I made the video a few pixels taller so I could add the time, date and other info in a bar along the bottom of the video. (The video went from 1296x972 to 1280x1008. I had to reduce the width due to #1608 but I don't think it's related.)
Since then, the playback stops randomly. I am playing two video streams at the same time (only one of which has had its size changed and started this problem), and when this bug is triggered, both streams break. Sometimes part of the picture from one video ends up showing in the other video's window, sometimes the video becomes very badly corrupted (turns green and magenta) but mostly the video just stops updating leaving only the last frame visible.
In order to fix it I have to exit both video players and restart - closing only one player is not enough. Sometimes only restarting one player will show no video, but sometimes it comes up with the last stuck frame still visible. Exiting all players and reloading them all gets this working again, but only for another few minutes until it crashes again.
I get various messages in the logs when this happens. Here is one exerpt:
I tried increasing the GPU mem as suggested. It was initially
gpu_mem=128which was fine until I enlarged the video, now I get the same messages even if I specifygpu_mem=384. I can't go large than this or the Pi won't boot. I havecma=300Mbut changing this value doesn't seem to do anything.System
cat /etc/rpi-issue)? Arch Linux ARM 64-bitvcgencmd version)?uname -a)?Linux 5.10.63-8-ARCH #1 SMP PREEMPT Thu Sep 16 14:24:31 UTC 2021 aarch64 GNU/LinuxThe text was updated successfully, but these errors were encountered: