RaspiCam V4L2 MJPEG mode stopped working under kernel 3.18

[nixnuex]

Before kernel 3.18 I could use the bcm2835-v4l2 kernel module to access the Raspberry Pi via /dev/video0 in MJPEG format with mjpg-streamer.
Starting with kernel 3.18 something seems to be broken, mjpg-streamer does not work in MJPEG mode anymore but only in YUYV mode.
dmsg output regarding this problem:

[...]
[   16.432994] media: Linux media interface: v0.10
[   16.467110] Linux video capture interface: v2.00
[   16.546597] bcm2835-v4l2: scene mode selected 0, was 0
[   16.553173] bcm2835-v4l2: V4L2 device registered as video0 - stills mode > 1280x720
[   16.579866] bcm2835-v4l2: Broadcom 2835 MMAL video capture ver 0.0.2 loaded.
[...]
[  160.383927] ------------[ cut here ]------------
[  160.384053] WARNING: CPU: 0 PID: 2427 at drivers/media/v4l2-core/videobuf2-core.c:2135 __vb2_queue_cancel+0xf4/0x160 [videobuf2_core]()
[  160.384072] Modules linked in: bcm2708_wdog ctr ccm bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media arc4 rt2800usb rt2800lib rt2x00usb rt2x00lib mac80211 cfg80211 crc_ccitt rfkill uio_pdrv_genirq uio
[  160.384181] CPU: 0 PID: 2427 Comm: mjpg_streamer Not tainted 3.18.3+ #739
[  160.384254] [<c00151fc>] (unwind_backtrace) from [<c0012710>] (show_stack+0x20/0x24)
[  160.384299] [<c0012710>] (show_stack) from [<c055599c>] (dump_stack+0x20/0x28)
[  160.384344] [<c055599c>] (dump_stack) from [<c0022edc>] (warn_slowpath_common+0x7c/0x9c)
[  160.384379] [<c0022edc>] (warn_slowpath_common) from [<c0022fb8>] (warn_slowpath_null+0x2c/0x34)
[  160.384437] [<c0022fb8>] (warn_slowpath_null) from [<bf1fca1c>] (__vb2_queue_cancel+0xf4/0x160 [videobuf2_core])
[  160.384525] [<bf1fca1c>] (__vb2_queue_cancel [videobuf2_core]) from [<bf1fe548>] (vb2_internal_streamoff+0x40/0xdc [videobuf2_core])
[  160.384598] [<bf1fe548>] (vb2_internal_streamoff [videobuf2_core]) from [<bf1fe6d8>] (vb2_streamoff+0x3c/0x60 [videobuf2_core])
[  160.384664] [<bf1fe6d8>] (vb2_streamoff [videobuf2_core]) from [<bf1fe74c>] (vb2_ioctl_streamoff+0x50/0x54 [videobuf2_core])
[  160.384862] [<bf1fe74c>] (vb2_ioctl_streamoff [videobuf2_core]) from [<bf1c1658>] (v4l_streamoff+0x28/0x2c [videodev])
[  160.385071] [<bf1c1658>] (v4l_streamoff [videodev]) from [<bf1c4af4>] (__video_do_ioctl+0x254/0x2e8 [videodev])
[  160.385279] [<bf1c4af4>] (__video_do_ioctl [videodev]) from [<bf1c449c>] (video_usercopy+0x180/0x560 [videodev])
[  160.385486] [<bf1c449c>] (video_usercopy [videodev]) from [<bf1c4898>] (video_ioctl2+0x1c/0x24 [videodev])
[  160.385674] [<bf1c4898>] (video_ioctl2 [videodev]) from [<bf1bf6b0>] (v4l2_ioctl+0x11c/0x14c [videodev])
[  160.385802] [<bf1bf6b0>] (v4l2_ioctl [videodev]) from [<c014c3f8>] (do_vfs_ioctl+0x3f0/0x5d4)
[  160.385843] [<c014c3f8>] (do_vfs_ioctl) from [<c014c620>] (SyS_ioctl+0x44/0x6c)
[  160.385881] [<c014c620>] (SyS_ioctl) from [<c000e8c0>] (ret_fast_syscall+0x0/0x48)
[  160.385900] ---[ end trace 156682ada5a6eb1e ]---

[popcornmix]

@6by9 any ideas?

[6by9]

Not off the top of my head. I seem to recall a fair amount changed around 3.14 within V4L2 buffer management (I only know that due to trying to build drivers for my DVB-S2 cards which the manufacturer hasn't upstreamed, just provides patches for particular kernel releases).

Oddly that callstack is showing it being solely within the V4L2 and videobuf2 framework code, it hasn't apparently touched the bcm2835-v4l2 driver at this point.

Any chance either of you could just try the simple streaming example using v4l2-ctl from http://www.raspberrypi.org/forums/viewtopic.php?f=43&t=62364 first post with each of the pixel formats (0 to 13 I think it is)? If not I'll try to find some time later on.

[6by9]

Hmm, the warning has the comment against it of
https://github.com/raspberrypi/linux/blob/rpi-3.18.y/drivers/media/v4l2-core/videobuf2-core.c#L2135

    /*
    * If you see this warning, then the driver isn't cleaning up properly
    * in stop_streaming(). See the stop_streaming() documentation in
    * videobuf2-core.h for more information how buffers should be returned
    * to vb2 in stop_streaming().
    */

The docs for stop_streaming then say

    * @stop_streaming: called when 'streaming' state must be disabled; driver
    * should stop any DMA transactions or wait until they
    * finish and give back all buffers it got from buf_queue()
    * callback by calling @vb2_buffer_done() with either
    * VB2_BUF_STATE_DONE or VB2_BUF_STATE_ERROR; may use
    * vb2_wait_for_all_buffers() function

Sounds plausible that we aren't returning all the buffers, although I need to check exactly when buffers are dequeued and requeued.
Seems a little strange that YUYV works when MJPEG doesn't as the buffer handling is no different. It may just be chance.

[6by9]

Further thought whilst waiting for my new 3.18 tree to build, this is only a warning message on stop_streaming so shouldn't stop anything else working. Do you get any frames through in MJPEG mode and things die when stopping, or does it just produce nothing at all?

[nixnuex]

Tested v4l2-ctl with pixelformat 0...13, no error messages during runs:

$ v4l2-ctl --list-formats
ioctl: VIDIOC_ENUM_FMT
    Index       : 0
    Type        : Video Capture
    Pixel Format: 'YU12'
    Name        : 4:2:0, packed YUV

    Index       : 1
    Type        : Video Capture
    Pixel Format: 'YUYV'
    Name        : 4:2:2, packed, YUYV

    Index       : 2
    Type        : Video Capture
    Pixel Format: 'RGB3'
    Name        : RGB24 (LE)

    Index       : 3
    Type        : Video Capture
    Pixel Format: 'JPEG' (compressed)
    Name        : JPEG

    Index       : 4
    Type        : Video Capture
    Pixel Format: 'H264' (compressed)
    Name        : H264

    Index       : 5
    Type        : Video Capture
    Pixel Format: 'MJPG' (compressed)
    Name        : MJPEG

    Index       : 6
    Type        : Video Capture
    Pixel Format: 'YVYU'
    Name        : 4:2:2, packed, YVYU

    Index       : 7
    Type        : Video Capture
    Pixel Format: 'VYUY'
    Name        : 4:2:2, packed, VYUY

    Index       : 8
    Type        : Video Capture
    Pixel Format: 'UYVY'
    Name        : 4:2:2, packed, UYVY

    Index       : 9
    Type        : Video Capture
    Pixel Format: 'NV12'
    Name        : 4:2:0, packed, NV12

    Index       : 10
    Type        : Video Capture
    Pixel Format: 'BGR3'
    Name        : RGB24 (BE)

    Index       : 11
    Type        : Video Capture
    Pixel Format: 'YV12'
    Name        : 4:2:0, packed YVU

    Index       : 12
    Type        : Video Capture
    Pixel Format: 'NV21'
    Name        : 4:2:0, packed, NV21

    Index       : 13
    Type        : Video Capture
    Pixel Format: 'BGR4'
    Name        : RGB32 (BE)

$ v4l2-ctl --set-fmt-video=width=1920,height=1088,pixelformat=0
$ v4l2-ctl --stream-mmap=3 --stream-count=100 --stream-to=pf0.264
<<<<< 1 fps
[...]

This produces files between 4 and 800 MB which I could not play on my Mac. For every run the warning appears in dmeg at the end of the stream (with different PIDs):

[  379.793424] ------------[ cut here ]------------
[  379.793570] WARNING: CPU: 0 PID: 2468 at drivers/media/v4l2-core/videobuf2-core.c:2135 __vb2_queue_cancel+0xf4/0x160 [videobuf2_core]()
[  379.793909] Modules linked in: bcm2708_wdog ctr ccm bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media arc4 rt2800usb rt2800lib rt2x00usb rt2x00lib mac80211 cfg80211 crc_ccitt rfkill uio_pdrv_genirq uio
[  379.794117] CPU: 0 PID: 2468 Comm: v4l2-ctl Not tainted 3.18.3+ #739
[  379.794191] [<c00151fc>] (unwind_backtrace) from [<c0012710>] (show_stack+0x20/0x24)
[  379.794259] [<c0012710>] (show_stack) from [<c055599c>] (dump_stack+0x20/0x28)
[  379.794305] [<c055599c>] (dump_stack) from [<c0022edc>] (warn_slowpath_common+0x7c/0x9c)
[  379.794362] [<c0022edc>] (warn_slowpath_common) from [<c0022fb8>] (warn_slowpath_null+0x2c/0x34)
[  379.794434] [<c0022fb8>] (warn_slowpath_null) from [<bf1fca1c>] (__vb2_queue_cancel+0xf4/0x160 [videobuf2_core])
[  379.794541] [<bf1fca1c>] (__vb2_queue_cancel [videobuf2_core]) from [<bf1fe548>] (vb2_internal_streamoff+0x40/0xdc [videobuf2_core])
[  379.794636] [<bf1fe548>] (vb2_internal_streamoff [videobuf2_core]) from [<bf1fe6d8>] (vb2_streamoff+0x3c/0x60 [videobuf2_core])
[  379.794707] [<bf1fe6d8>] (vb2_streamoff [videobuf2_core]) from [<bf1fe74c>] (vb2_ioctl_streamoff+0x50/0x54 [videobuf2_core])
[  379.794926] [<bf1fe74c>] (vb2_ioctl_streamoff [videobuf2_core]) from [<bf1c1658>] (v4l_streamoff+0x28/0x2c [videodev])
[  379.795362] [<bf1c1658>] (v4l_streamoff [videodev]) from [<bf1c4af4>] (__video_do_ioctl+0x254/0x2e8 [videodev])
[  379.795615] [<bf1c4af4>] (__video_do_ioctl [videodev]) from [<bf1c449c>] (video_usercopy+0x180/0x560 [videodev])
[  379.795853] [<bf1c449c>] (video_usercopy [videodev]) from [<bf1c4898>] (video_ioctl2+0x1c/0x24 [videodev])
[  379.796070] [<bf1c4898>] (video_ioctl2 [videodev]) from [<bf1bf6b0>] (v4l2_ioctl+0x11c/0x14c [videodev])
[  379.796220] [<bf1bf6b0>] (v4l2_ioctl [videodev]) from [<c014c3f8>] (do_vfs_ioctl+0x3f0/0x5d4)
[  379.796484] [<c014c3f8>] (do_vfs_ioctl) from [<c014c620>] (SyS_ioctl+0x44/0x6c)
[  379.796531] [<c014c620>] (SyS_ioctl) from [<c000e8c0>] (ret_fast_syscall+0x0/0x48)
[  379.796551] ---[ end trace f48dbff5810843d2 ]---

Tested again mjpeg-streamer and it works without problems in YUYV mode (the -y flag, no warning in dmesg) but does not show any video in MJPEG mode (with warning in dmesg):

$ ./mjpg_streamer -i "./input_uvc.so -n -y" -o "./output_http.so -n -w ./www"
MJPG Streamer Version: svn rev: 
 i: Using V4L2 device.: /dev/video0
 i: Desired Resolution: 640 x 480
 i: Frames Per Second.: -1
 i: Format............: YUYV
 i: JPEG Quality......: 80
 i: TV-Norm...........: DEFAULT
 o: www-folder-path...: ./www/
 o: HTTP TCP port.....: 8080
 o: username:password.: disabled
 o: commands..........: disabled
^Csetting signal to stop
 i: cleaning up ressources allocated by input thread
force cancellation of threads and cleanup resources
 o: cleaning up ressources allocated by server thread #00
done

$ ./mjpg_streamer -i "./input_uvc.so -n" -o "./output_http.so -n -w ./www"
MJPG Streamer Version: svn rev: 
 i: Using V4L2 device.: /dev/video0
 i: Desired Resolution: 640 x 480
 i: Frames Per Second.: -1
 i: Format............: JPEG
 i: TV-Norm...........: DEFAULT
 o: www-folder-path...: ./www/
 o: HTTP TCP port.....: 8080
 o: username:password.: disabled
 o: commands..........: disabled
^Csetting signal to stop
 i: cleaning up ressources allocated by input thread
force cancellation of threads and cleanup resources
 o: cleaning up ressources allocated by server thread #00
done

If it's just a warning, it's strange that things stopped working with mjpg-streamer just by updating to 3.18. Can I help with something else?

[6by9]

Thank you so much for doing that - my time for playing with the Pi is very limited, so having a decent description and as much debug as possible makes my life much easier.

Pixel format 4 (H264) should play back in VLC happily.
Format 5 (MJPG) will if you push it into a container first (some runes with avconv that I can't quite remember, and the forum search isn't bringing up for me. VLC won't take a raw MJPEG elemental stream in the same way it will with H264).
The rest are unlikely to as they are raw pixel data and have no headers to tell a decoder how to interpret the bytes.

I'm still fairly confident that is just a warning and shouldn't cause any issues. It's also on stop_streaming only, and I wouldn't expect mjpg_streamer to be stopping the stream too often.

There is extra logging you can enable from the V4L2 driver via debugfs. I can never remember exactly where it is, but a find -name bcm2835_v4l2_debug from / should show it up, and you can then do
echo 2 > <path>/bcm2835_v4l2_debug
and you'll find that a load more info gets dumped into dmesg.

I'm looking at a couple of other minor issues in the time that I do get, but should hopefully get a chance to look at this properly soon.

[nixnuex]

Okay, happy if I can do my part. Will try the next steps later, there seems to be something else regarding mjpg-streamer and 3.18, but with a USB webcam: http://www.raspberrypi.org/forums/viewtopic.php?f=28&t=97983

[6by9]

That does seem to imply that something odd is happening to an MJPEG stream, potentially within the V4L2 core. I can't see why both Pi Camera and a Logitech camera would suddenly start showing issues otherwise.

That might make capturing the output from v4l2-ctl with pixelformat=5 and analysing it more useful. I think the runes need to be something like
avconv -i in.mjpg -c:v copy out1.mp4
but I may be wrong, and I don't have a test file here to confirm it with.

[nixnuex]

Yes, the output pixelformat=5 (MJPG) plays back without problems and shows the video as expected. So indeed, that part seems to work. Whatever causes the warning and stops mjpg-streamer from working remains unknown.

[6by9]

I did download and run the latest V4L2 conformance test last night. It threw up 7 failures (about 38 passes) which may shed some light on the issue. Generally they're minor whinges rather doing anything significantly wrong though.
I'll hopefully get a couple of hours over the weekend to play.

It'd also be worth trying qv4l2 (part of v4l-utils same as v4l2-ctl) which is a QT graphical app that provides the equivalent to v4l2-ctl but also allows viewing the images. If MJPEG works from there, then I really am running out of ideas.

[6by9]

May be a total red herring, but just picked up on Hexxeh/rpi-firmware#57 which has been linked to https://bugzilla.kernel.org/show_bug.cgi?id=81611 "[regression] in kernel 3.16, uvcvideo no longer works for usb Laptop_Integrat..."
A change in v4l2-core.c is resetting bytesused to 0 whenever the buffer is queued, but the app (uvcvideo in their case) was then processing the buffer and expecting that value to be still set. I wonder if mjpeg-streamer is making the same assumption for MJPEG but not for YUYV.

[mirak011]

I did some testing with my Logitech C270 and following apps: guvcview, mjpg-streamer and qv4l2 on Rasbian with kernel 3.12 and 3.18 and also on my desktop with Ubuntu 14.04 with kernel 3.13 and 3.16. All application are running OK with kernel 3.12 (raspberry) and 3.13 (desktop), however with newer kernel 3.16 (desktop) and 3.18 (raspberry) only qv4l2 is working correctly and mjpg-streamer is working only with YUYV, but not with MJPG.
On the other hand my second raspberry pi and raspberry cam is working well also with kernel 3.18 with specific version of mjpg-streamer for raspberry cam - https://github.com/jacksonliam/mjpg-streamer. It might really indicate a bug in original version of mjpg-streamer and guvcview that has been revealed due to change in v4l2-core.c

[Whobeu]

Yes this does make sense. Per a discussion in http://www.mail-archive.com/[email protected]/msg80144.html between the developers it seems the v4l2-core.c module had a bug fixed that gucview and, apparently, mjpg-streamer relied on. With the "fix", the apps broke. It looks like Paulo Assis did develop a workaround to the fix in gucview but I assume the input_uvc.c module has not been touched. I am going to test the "jacksonliam" fork and see how that works.

UPDATE: I looked at the "jacksonliam" project and there is nothing in the input_uvc plugin that has been updated recently. Not sure why it should work under 3.18+ without compensating for the bytesused field change. I'm going to skip testing it at this point.

[6by9]

https://github.com/jacksonliam/mjpg-streamer is using the MMAL library directly rather than going through UVC (not applicable for the Pi Camera anyway) or V4L2, hence probably working without issue with the Pi Camera.

[mirak011]

I think mjpg-streamer can use V4L2, because I compiled with "make USE_LIBV4L2=true"

[6by9]

mjpg_streamer by default uses V4L2, and USE_LIBV4L2 builds the UVC input module with different options to use the libv4l2 helper library.

jacksonliam's version includes an alternate input plugin to talk directly to the Pi GPU using MMAL. You didn't say what command line you were using to launch mjpg-streamer in each case. It looks like you have to amend the command line to ask for input-raspicam to get the direct MMAL option, so perhaps it was still using V4L2 but somehow not hitting the issue.

[mirak011]

Sorry for confusion, I didn't noticed you were referring only to "jacksonliam's version", so my comment doesn't make sense. (I used "make USE_LIBV4L2=true" only with general version of mjpg-streamer.)

Here is a patch for mjpg-streamer as was published on http://www.raspberrypi.org/forums/viewtopic.php?f=28&t=97983&sid=dfb94ec454719c686940ff9983162fba

--- plugins/input_uvc/input_uvc.c       (revision 174)
+++ plugins/input_uvc/input_uvc.c       (working copy)
@@ -405,9 +405,13 @@
         if(pcontext->videoIn->formatIn == V4L2_PIX_FMT_YUYV) {
             DBG("compressing frame from input: %d\n", (int)pcontext->id);
             pglobal->in[pcontext->id].size = compress_yuyv_to_jpeg(pcontext->videoIn, pglobal->in[pcontext->id].buf, pcontext->videoIn->framesizeIn, gquality);
+            /* copy this frame's timestamp to user space */
+            pglobal->in[pcontext->id].timestamp = pcontext->videoIn->buf.timestamp;
         } else {
             DBG("copying frame from input: %d\n", (int)pcontext->id);
-            pglobal->in[pcontext->id].size = memcpy_picture(pglobal->in[pcontext->id].buf, pcontext->videoIn->tmpbuffer, pcontext->videoIn->buf.bytesused);
+            pglobal->in[pcontext->id].size = memcpy_picture(pglobal->in[pcontext->id].buf, pcontext->videoIn->tmpbuffer, pcontext->videoIn->tmpbytesused);
+            /* copy this frame's timestamp to user space */
+            pglobal->in[pcontext->id].timestamp = pcontext->videoIn->tmptimestamp;
         }

 #if 0
@@ -418,8 +422,6 @@
         prev_size = global->size;
 #endif

-        /* copy this frame's timestamp to user space */
-        pglobal->in[pcontext->id].timestamp = pcontext->videoIn->buf.timestamp;

         /* signal fresh_frame */
         pthread_cond_broadcast(&pglobal->in[pcontext->id].db_update);
Index: plugins/input_uvc/v4l2uvc.c
===================================================================
--- plugins/input_uvc/v4l2uvc.c (revision 174)
+++ plugins/input_uvc/v4l2uvc.c (working copy)
@@ -450,6 +450,8 @@
         */

         memcpy(vd->tmpbuffer, vd->mem[vd->buf.index], vd->buf.bytesused);
+        vd->tmpbytesused = vd->buf.bytesused;
+        vd->tmptimestamp = vd->buf.timestamp;

         if(debug)
             fprintf(stderr, "bytes in used %d \n", vd->buf.bytesused);
Index: plugins/input_uvc/v4l2uvc.h
===================================================================
--- plugins/input_uvc/v4l2uvc.h (revision 174)
+++ plugins/input_uvc/v4l2uvc.h (working copy)
@@ -28,6 +28,7 @@


 #include <stdio.h>
+#include <stdint.h>
 #include <string.h>
 #include <fcntl.h>
 #include <unistd.h>
@@ -105,6 +106,8 @@
     int framecount;
     int recordstart;
     int recordtime;
+    uint32_t tmpbytesused;
+    struct timeval tmptimestamp;
 };

 /* context of each camera thread */

[6by9]

Can we close this then, as it isn't a firmware bug?
I'll create a new (kernel) issue so I don't forget to look into the warning on stop_streaming.

[Whobeu]

IMHO it is fixed. Streaming is working fine on 3.18.xx with the above patches.