MMAL hungs if camera preview port encoding is not OPAQUE with capture port enabled

[Terminus-IMRC]

It seems that MMAL hungs the system if the encoding of camera preview port is not OPAQUE with capture port enabled.

Please try with this code: https://github.com/Terminus-IMRC/mmal-hungs-if-preview-encoding-is-not-opaque
This program creates vc.ril.camera, vc.ril.render and vc.null_sink components and connect them in this way:

camera[0] -- [0]render
camera[2] -- [0]null

So the preview port of the camera is displayed on screen and the still port of the camera is consumed by null.

If you leave these things, MMAL freezes the entire system.

• ENCODING_PREVIEW to an encoding other than OPAQUE such as RGB24, RGBA and so on.
• CAPTURE to 1 (MMAL_TRUE).

If you change ENCODING_PREVIEW to OPAQUE or CAPTURE to 0 (MMAL_FALSE), it will work.

I think I'm not wrong in usage of the API.
Thanks for reading.

[6by9]

null_sink doesn't support OPAQUE as produced by camera->output[2]. null_sink never returns the buffer indicating that the data has been consumed, therefore the camera stalls waiting for the capture to complete. That really ought to be fixed up and rejected in a more sensible manner, but no data is exchanged between the ports until the first buffer is passed, and the encoding doesn't make it explicit.
Switch to I420 and that goes away.

ZERO_COPY should do nothing with a mmal_connection with MMAL_CONNECTION_FLAG_TUNNELLING - the buffers then all stay on the VPU, so there is no need to copy between ARM and VC memory.

The fact that you're managing to trigger a kernel panic on both my Pi 2 & 3 is more entertaining. I was having issues last week anyway, so that may be totally unrelated and just mean I need to take a clean image.

[Terminus-IMRC]

Thank you for your response. I removed ZERO_COPY settings and changed ENCODING_CAPTURE to I420 on the repo. Then the firmware continues working and the kernel stops, as your pis do. I'm using CM1+{OV5647,IMX219} and the latest Hexxeh/rpi-firmware.

[6by9]

OK, got it. Firmware bug.

If the preview port is set up for an RGB or YUYV format the ISP gets incorrectly programmed on the still capture and will start scribbling data at physical address 0x00000000 :-o
Selecting OPAQUE, I420, or NV12 should be safe at the moment. I'll sort a firmware fix for the other formats.

[Terminus-IMRC]

Thank you!

[6by9]

Patch sorted and pushed internally.
I don't know exactly when it'll get picked up, but should be fairly soon.

[6by9]

Fix for ENCODING_PREVIEW being other than I420 or NV12 has been released.
Please use rpi-update (usual caution to be observed) and retest.

I haven't tested ZERO_COPY with both settings but it probably makes no difference in this case, just it is redundant.

I've just raised #390 to note the OPAQUE handling from camera->output[2] to null_sink, so if your use case is now working then please close this issue.

[Terminus-IMRC]

It seems to be fixed. Thank you!

[6by9]

I'm finally getting a chance to look into this one in more detail.

vc.null_sink I hadn't even realised was built and shipped - I'd read it as vc.ril.null_sink.
vc.ril.null_sink now supports opaque buffers (input[0]=audio, input[1]=video, input[2]=video).

There's a second fix for the framework that means the piepeline won't lock up if connected to camera->output[2] with opaque buffers regardless of sink component, but they won't necessarily encode the correct thing - no real way around that as the two formats don't necessarily intersect in requirements.