Oops with vchiq_prepare_bulk_data (tc358743 + OMX JPEG encoder) #4669
Comments
mdevaev
changed the title
|
I have updated the description and localized the problem. It occurs when we try to use OMX JPEG encoding when capturing video from TC358743. |
|
The crash logs suggest the failures are the consequences of memory corruption. |
|
It doesn't show up on 5.10.52, maybe it's some new bug? On the new kernel, failure occurs 100% of the time when using OMX. |
|
Tripped when encoding JPEG with ustreamer Kernel oops Edit: Firmware looks to have died, but hasn't flushed the cache so |
Yep - that looks like (black) pixel data in a place where no pixel data should be. |
|
Rebuilt my kernel at 2a29770 (5.10.77) and top of tree firmware (with a few local mods). Crashed. Reset my firmware back ("17 and 10 different commits each, respectively", top commit from origin is bcdfe55d5). I'll bisect it. |
|
Working at 1920x1080 seems to trigger things more frequently. (I was using 1280x720 before). I've just had a logged as well, which has also killed everything. What I thought was working with 5.10.77 and my old firmware does fail at 1080p. I'm reverting back to the firmware from 5.10.52, but one thought I do have is that the buffers allocated by bcm2835-unicam and consumed by the H264 encoder do NOT have to be aligned to a multiple of 16 lines, whilst the IL JPEG DOES need 16 line alignment. It's not that you're asking it to copy more data than is allocated? Firmware from 5.10.52 has just crashed with the kernel logging |
|
|
|
raspberrypi/rpi-firmware@c59a637 seems to be the first to really trip this, so it would imply something changed with #4641. Slightly odd as that didn't touch memory allocation of buffer usage, but there you go. |
|
@6by9 I have already encountered alignment problems and spied code in yavta and gstreamer. At least it worked before the update. |
|
Please let me know if I can help with testing or anything else. |
|
By the way, how would you like the new V4L2 M2M interface to support JPEG encoding? Since you're throwing away proprietary libraries, it would be great to have a replacement for this feature. My idea was to make /dev/video21 for ril.image_encode. I will be happy to make draft implementation and patch if you are interested in taking this upstream. I saw that libcamera uses libjpeg to encode images and on the RPi forum I was offered to use encoding on cores, but as you saw, 1080p encoding consumes a huge amount of CPU resources and takes more time, which is a disaster for PiKVM, we really need hardware encoding to keep zero latency. PS: I'm sorry to ask this here. It's just that if M2M works and is accepted into the RPi upstream, I can forget about the problems of OMX :) Unless the crash described is related to something lower-level and will still require a fix. |
|
The encoder role in bcm2835-codec already supports MJPEG via video_encode. image_encode messes around with EXIF headers and similar, which are all irrelevant under V4L2 as there are no controls to pass them in. All V4L2 is really after is the basic encoded image, which MJPEG gives you. It also does rate control rather than manually messing with quantisation parameters. I realise that you were running multiple image_encode components in parallel as feeding in YUYV (or similar) requires a format conversion first, so effectively you were queuing those up. The structure of the MJPEG codec may not give the required performance there, but I couldn't without trying it. There is an optimisation that could be made for MJPEG if it was close. V4L2 does avoid VCHIQ bulk transfers, so should avoid this issue. |
|
Thanks for the explanation. I couldn't get MJPEG from encoder in version 5.10.52, did it appear later? Whatever encoding parameters I used, it still turned out to be H264. v4l2-ctl --list-formats also showed that /dev/video11 supports only H264. As for the speed - in any case, I'll try and compare. Even if V4L2 is a bit slower, I expect a speed boost (or compensation) due to the possibility of using DMA, since with OMX I couldn't make it work. |
|
Okay, it really is. Even if I set the capture buffer format as V4L2_PIX_FMT_MJPEG, the encoder still produces H264. This is even true on new kernels. How do I achieve MJPEG? What parameters should I use? Here I made a special version of ustreamer (m2m branch). To run, it's enough to simply use https://github.com/pikvm/ustreamer/blob/m2m/src/ustreamer/m2m.c#L152 |
|
which certainly looks like JPEG / MJPEG to me. H264 would start with a start code (00 00 00 01). If the driver doesn't like something about the format passed with VIDIOC_S_FMT, the spec requires it to correct it to something that it does like and return that. It is not permitted to fail the VIDIOC_S_FMT call unless the type is invalid. |
|
Maybe something is wrong with my code, because even judging by the start bytes it produces H264. I'll build a gstreamer and try it. |
|
Okay, in my case, v4l2jpegenc is unavailable, since apparently gst checks the capabilities of the encoder and disables the unavailable component. At the same time, I have v4l2jpegdec and v4l2h264enc available. Do I need some kind of latest firmware version? Or is it enabled by some kernel options? |
|
I was just using a Bullseye image and checked with gst-inspect which components starting v4l2 existed. IIRC Bullseye is using GStreamer 1.18. |
|
1.18 too, but I'm using Arch. I understand that the problem is probably in my kernel, I just want to figure out what exactly I have to enable/tune/modify in it to get a jpeg. |
|
It's very strange. I just installed Bullseye and tried gstreamer. The result is the same as for Arch. Maybe the problem is something specific to a particular revision of the board? |
|
I tested fresh Bullseye on Raspberry Pi 4 with 2, 4 and 8 Gb RAM with 32 bit. I also tried 4 Gb and 64 bit. I tried the kernel from the repository and from rpi-update. JPEG encoder was not available anywhere. Maybe I should load some modules or use special kernel parameters? |
|
I found out that PS I'm exploring jpeg encoder a bit more and have completely migrated my software. Then OMX issues can be closed with wontfix %) |
|
Encoding used to be only available when used with the camera, hence start_x=1 (or start_debug=1). H264 encode obviously got added to the base image at some point. |
|
Maybe it's worth adding mpeg encoding in the base image as well as h264? Just because using this for tc358743 is a fairly common case, and it would be strange to include start_x for the sake of it. |
|
The JPEG encoder is working, but I can't change the image quality in any way - it's terrible. I made a patch to configure MMAL_PARAMETER_JPEG_Q_FACTOR, but it didn't affect anything at all.
Interestingly, gstreamer also gives such a picture:
Should I configure something else so that MMAL doesn't ignore the q-factor? Maybe this is due to the fact that MJPG/ril.video_encode, not JPEG/ril.image_encode? |
|
MJPEG is configured via bitrate, not via a Q factor. |
|
Your invalid length issue sort of confirms that you aren't handling buffer sizes correctly. image_encode requires the height to be aligned to a multiple of 16 lines. The Unicam driver doesn't. You can use |
My code already takes into account the need for alignment, and this works fine with V4L2_MEMORY_MMAP. I used the same logic that is already implemented for alignment/cropping with DECODER role. Is that not enough?
Ideally, I would like the userspace code not to know about the specifics of working with buffers. PS: Your idea of multiplying by -1 did not help, and there is another problem with DMA for video_encode, for which I made a separate issue. |
My code already takes into account the need for alignment, and this works fine with V4L2_MEMORY_MMAP. I used the same logic that is already implemented for alignment/cropping with DECODER role. Is that not enough?
Ideally, I would like the userspace code not to know about the specifics of working with buffers. PS: Your idea of multiplying by -1 did not help, and there is another problem with DMA for video_encode, for which I made a separate issue. |
|
REQBUFS will give you a buffer that is the size that that V4L2 device wants. When importing DMABUFs the kernel can check the size of the underlying allocation to ensure that userspace doesn't allocate X, and then ask another subsystem to import X+Y. That's why it complains. JPEG and H264 encoding natively need to work in macroblocks, hence frequently having the requirement for the height (and stride) to be a multiple of 16. |
|
I see, thank you for explanation. Could you also say something about the DMA/video_encode crash and the second problem with qfactor? |
|
@6by9 sup? |
|
@6by9 Could we somehow solve the problem with IJG because negative values don't work, and this is a blocker for our further work related to the migration to Bullseye. |
|
Firmware patch created adding MMAL_PARAMETER_JPEG_IJG_SCALING. |
|
Thank you! I'll check it. |
|
Will the definition of the |
kernel: Patching lan78xx for SOF_TIMESTAMPING_TX_SOFTWARE support See: raspberrypi/linux#4856 kernel: i2c: bcm2835: Make clock-stretch timeout configurable See: raspberrypi/linux#4855 kernel: drm/vc4: Add DRM 210101010 RGB formats for hvs5. See: raspberrypi/linux#4859 kernel: vc4-kms-dpi overlay updates See: raspberrypi/linux#4860 kernel: Add Support for the Geekworm MZP280 DPI Display See: raspberrypi/linux#4853 kernel: DRM: Clean up handling of panel orientation See: raspberrypi/linux#4862 kernel: Add support for the MAX30102 heart rate and blood oxygen sensor See: raspberrypi/linux#4535 firmware: mmal: Add mapping for IL OMX_IndexParamBrcmEnableIJGTableScaling param See: raspberrypi/linux#4669 userland: Handle overlay parameters embedded in overlay_map.dtb See: raspberrypi/linux#4860
kernel: Patching lan78xx for SOF_TIMESTAMPING_TX_SOFTWARE support See: raspberrypi/linux#4856 kernel: i2c: bcm2835: Make clock-stretch timeout configurable See: raspberrypi/linux#4855 kernel: drm/vc4: Add DRM 210101010 RGB formats for hvs5. See: raspberrypi/linux#4859 kernel: vc4-kms-dpi overlay updates See: raspberrypi/linux#4860 kernel: Add Support for the Geekworm MZP280 DPI Display See: raspberrypi/linux#4853 kernel: DRM: Clean up handling of panel orientation See: raspberrypi/linux#4862 kernel: Add support for the MAX30102 heart rate and blood oxygen sensor See: raspberrypi/linux#4535 firmware: mmal: Add mapping for IL OMX_IndexParamBrcmEnableIJGTableScaling param See: raspberrypi/linux#4669 userland: Handle overlay parameters embedded in overlay_map.dtb See: raspberrypi/linux#4860
|
It's pushed to rpi-update (but that doesn't affect kernel tree). |
|
Added to userland - raspberrypi/userland@8fa944c Until it's used in the kernel there is little point in adding it there. |
|
Got it |
|
@6by9 Something doesn't seem to be working. I fixed my patch to the latest kernel version and installed a new firmware. The return value of changing the IJG param is -3: u32 enable = 1;
int x = vchiq_mmal_port_parameter_set(ctx->dev->instance,
&ctx->component->output[0],
MMAL_PARAMETER_JPEG_IJG_SCALING,
&enable,
sizeof(enable));MMAL_PARAMETER_JPEG_IJG_SCALING is 0x10061, same as userspace header (checked). Also tried Am I doing something wrong? |
- firmware: ldconfig: Discard subsequent chunks from a truncated line See: #1669 - firmware: cec: Fail set_passive_mode when running with kms - firmware: Firmware: Remove PWM/audio traits for CM4 - firmware: usb: Fix non-BCM2711 MSD support See: raspberrypi/usbboot#102 - firmware: arm-loader: Fix kernel8.img selection on 2837 with arm_64bit=1 See: #1671 - firmware: improve firmware camera detection - firmware: arm_loader: Load vl805 overlay on CM4 See: https://forums.raspberrypi.com/viewtopic.php?t=326088 - firmware: gencmdserv: Add mailbox interface to gencmd - firmware: arm_loader: Only clip min/max to the same value for turbo clocks - firmware: dtoverlay: Don't mix non-fatal errors and offsets See: #1686 - firmware: platform: Limit max clock-id to CLOCK_VEC for now See: #1688 - firmware: mmal: Add mapping for IL OMX_IndexParamBrcmEnableIJGTableScaling param See: raspberrypi/linux#4669
|
Sup? |
|
I had got the mapping wrong and hadn't passed the port parameter through. Firmware patch being reviewed now. sets the right flag in the firmware, and it is then passed into the encoder. |
|
Thank you! I will be waiting for the firmware update. |
|
Any news about it? When will this fix be released? |
|
I'm not sure what the holdup with the firmware is, but there should be a new release today or tomorrow. |
|
rpi-firmware has been bumped - |
|
It seems it's working now, thank you. The next question: could you enable image encoder without |
Uh oh!
There was an error while loading. Please reload this page.
Describe the bug
The latest version of 5.10 kernel crashing when using OMX JPEG encoding and TC358743 with 720p video source resolution.
To reproduce
Take TC358743 and ustreamer.
Add
cma=128Mto cmdline.txt andgpu_mem=256to config.txt and try this command:You will see the crash.
System
Raspberry Pi 4 B+ 4Gvcgencmd version:Linux pikvm 5.10.76-4-raspberrypi4-ARCH #1 SMP Fri Nov 5 08:11:58 MSK 2021 armv7l GNU/LinuxLogs
OOPS 1
OOPS 2
The text was updated successfully, but these errors were encountered: