ffmpeg h264_v4lm2m encoding from jpgs fails. "Encoder requires yuv420p pixel format."

[JooJooBee666]

Describe the bug

Let me preface by saying this is a patched version of ffmpeg from [https://github.com/jc-kynesim/rpi-ffmpeg.git branch test/4.3.4/rpi_main](https://github.com/jc-kynesim/rpi-ffmpeg.git branch test/4.3.4/rpi_main)
Given a set of numbered jpgs, one would expect the following command to create a playable video:
ffmpeg -r 5 -i ./frame%d.jpg -c:v h264_v4l2m2m temp.mp4
However, this is the output:

ffmpeg version pi/4.3.4/rpi_14-3-ge6ce03a5e4-rpi Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/home/pi/source/rpi-ffmpeg/out/arm64-bullseye-4.3.4-shared-rel/install --libdir=/home/pi/source/rpi-ffmpeg/out/arm64-bullseye-4.3.4-shared-rel/install/lib/aarch64-linux-gnu --incdir=/home/pi/source/rpi-ffmpeg/out/arm64-bullseye-4.3.4-shared-rel/install/include/aarch64-linux-gnu --disable-stripping --disable-thumb --enable-v4l2-request --enable-libdrm --enable-vout-egl --enable-vout-drm --enable-shared --disable-mmal --enable-sand --enable-libx264 --enable-gpl --extra-cflags='-ggdb ' --extra-cxxflags=' ' --extra-ldflags= --extra-libs= --extra-version=rpi
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, image2, from './frame%d.jpg':
  Duration: 00:00:00.20, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0x55c654a450]  <<< v4l2_encode_init: fmt=12/-1
[h264_v4l2m2m @ 0x55c654a450] Using device /dev/video11
[h264_v4l2m2m @ 0x55c654a450] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x55c654a450] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x55c654a450] Encoder requires yuv420p pixel format.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

I then tried the following: ffmpeg -pix_fmt yuv420p -r 5 -i ./frame%d.jpg -c:v h264_v4l2m2m temp.mp4 but again this failed, but with another error (though functionally the same):

[h264_v4l2m2m @ 0x55833ed500]  <<< v4l2_encode_init: fmt=12/-1
[h264_v4l2m2m @ 0x55833ed500] Using device /dev/video11
[h264_v4l2m2m @ 0x55833ed500] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x55833ed500] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x55833ed500] Encoder requires yuv420p pixel format.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

However, using the libx264 encoder works just fine:

ffmpeg -y -pix_fmt yuv420p -r 5 -i ./frame%d.jpg -c:v libx264 temp.mp4
ffmpeg version pi/4.3.4/rpi_14-3-ge6ce03a5e4-rpi Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/home/pi/source/rpi-ffmpeg/out/arm64-bullseye-4.3.4-shared-rel/install --libdir=/home/pi/source/rpi-ffmpeg/out/arm64-bullseye-4.3.4-shared-rel/install/lib/aarch64-linux-gnu --incdir=/home/pi/source/rpi-ffmpeg/out/arm64-bullseye-4.3.4-shared-rel/install/include/aarch64-linux-gnu --disable-stripping --disable-thumb --enable-v4l2-request --enable-libdrm --enable-vout-egl --enable-vout-drm --enable-shared --disable-mmal --enable-sand --enable-libx264 --enable-gpl --extra-cflags='-ggdb ' --extra-cxxflags=' ' --extra-ldflags= --extra-libs= --extra-version=rpi
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, image2, from './frame%d.jpg':
  Duration: 00:00:00.20, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x559125a520] using SAR=1/1
[libx264 @ 0x559125a520] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x559125a520] profile High, level 4.0, 4:2:0, 8-bit
[libx264 @ 0x559125a520] 264 - core 160 r3011 cde9a93 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=5 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'temp.mp4':
  Metadata:
    encoder         : Lavf58.45.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 5 fps, 10240 tbn, 5 tbc
    Metadata:
      encoder         : Lavc58.91.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
[swscaler @ 0x559153b110] deprecated pixel format used, make sure you did set range correctly
frame=    4 fps=2.1 q=-1.0 Lsize=    1088kB time=00:00:00.20 bitrate=44531.9kbits/s speed=0.103x    
video:1087kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.080777%
[libx264 @ 0x559125a520] frame I:1     Avg QP:22.94  size:390521
[libx264 @ 0x559125a520] frame P:3     Avg QP:22.65  size:240579
[libx264 @ 0x559125a520] mb I  I16..4:  0.5% 99.1%  0.4%
[libx264 @ 0x559125a520] mb P  I16..4:  2.1% 63.4%  5.8%  P16..4:  7.0% 10.5% 10.8%  0.0%  0.0%    skip: 0.3%
[libx264 @ 0x559125a520] 8x8 transform intra:92.1% inter:82.8%
[libx264 @ 0x559125a520] coded y,uvDC,uvAC intra: 89.9% 81.4% 25.6% inter: 87.5% 46.9% 0.2%
[libx264 @ 0x559125a520] i16 v,h,dc,p: 21% 24% 14% 41%
[libx264 @ 0x559125a520] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 14% 18% 23%  5%  8%  6% 10%  6% 10%
[libx264 @ 0x559125a520] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 24% 12%  6%  8%  6%  9%  4%  7%
[libx264 @ 0x559125a520] i8c dc,h,v,p: 45% 28% 22%  5%
[libx264 @ 0x559125a520] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x559125a520] ref P L0: 95.8%  1.8%  2.3%  0.0%
[libx264 @ 0x559125a520] kb/s:11122.57

Steps to reproduce the behaviour

See above

Device (s)

Raspberry Pi 4 Mod. B

System

Raspberry Pi reference 2022-09-06
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 827affcc11aaf7aa577d15daf02fb40b64392380, stage4

Aug 26 2022 14:03:16
Copyright (c) 2012 Broadcom
version 102f1e848393c2112206fadffaaf86db04e98326 (clean) (release) (start)

Linux pi5 5.15.61-v8+ #1579 SMP PREEMPT Fri Aug 26 11:16:44 BST 2022 aarch64 GNU/Linux

4b with 8GB RAM

Logs

No response

Additional context

I am ultimately trying to use an ffmpeg pipe to encode a stream of jpeg frames in a Python application. However, when I do that ffmpeg throws nasty messages at me. I decided to just simplify and target encoding some already created jpeg images to narrow down the issues. Clearly, this is a problem with the h264_v4l2m2m encoder as the SAME exact command works fine when the encoder is switched to libx264. Also, specifying additional parameters such as width, height and bitrate get rid of those specific errors, but the root "Encoder requires yuv420p pixel format" error is still the issue here.

[MaoJianwei]

Me too. How to resolve it?

when I run:

ffmpeg -y -s 320x240 -i /dev/video0 -vcodec h264_v4l2m2m -f flv rtmp://xxxxxx.maojianwei.com:xxxx/xxx/xxx

I got these:

[video4linux2,v4l2 @ 0xed9cf0] Time per frame unknown
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 286.983401, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 320x240, 31 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0xede5e0]  <<< v4l2_encode_init: fmt=13/-1
[h264_v4l2m2m @ 0xede5e0] Using device /dev/video11
[h264_v4l2m2m @ 0xede5e0] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0xede5e0] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0xede5e0] Encoder requires yuv420p pixel format.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

[6by9]

@JooJooBee666
FFmpeg's v4l2_m2m layer doesn't have support for AV_PIX_FMT_YUVJ420P

Looking at the definition for them https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/pixfmt.h#L78-L79

     AV_PIX_FMT_YUVJ420P,  ///< planar YUV 4:2:0, 12bpp, full scale (JPEG), deprecated in favor of AV_PIX_FMT_YUV420P and setting color_range
    AV_PIX_FMT_YUVJ422P,  ///< planar YUV 4:2:2, 16bpp, full scale (JPEG), deprecated in favor of AV_PIX_FMT_YUV422P and setting color_range

Using AV_PIX_FMT_YUV420P with color_range would probably work on v4l2_m2m, but they appear not to have converted the code over in the nearly 13 years since they deprecated the format (FFmpeg/FFmpeg@98aea87).

Force the format on the input to the codec and it works but invokes a software conversion.

ffmpeg -r 5 -i ./frame%d.jpg -pix_fmt yuv420p -c:v h264_v4l2m2m temp.mp4

@MaoJianwei The codec doesn't support yuvj422p or yuv422p.
From the V4L2 level, the supported formats are

pi@pi:~/jpegs $ v4l2-ctl --list-formats-out -d 11
ioctl: VIDIOC_ENUM_FMT
	Type: Video Output Multiplanar

	[0]: 'YU12' (Planar YUV 4:2:0)
	[1]: 'YV12' (Planar YVU 4:2:0)
	[2]: 'NV12' (Y/UV 4:2:0)
	[3]: 'NV21' (Y/VU 4:2:0)
	[4]: 'RGBP' (16-bit RGB 5-6-5)
	[5]: 'RGB3' (24-bit RGB 8-8-8)
	[6]: 'BGR3' (24-bit BGR 8-8-8)
	[7]: 'AB24' (32-bit RGBA 8-8-8-8)
	[8]: 'BGR4' (32-bit BGRA/X 8-8-8-8)
	[9]: 'YUYV' (YUYV 4:2:2)
	[10]: 'YVYU' (YVYU 4:2:2)
	[11]: 'UYVY' (UYVY 4:2:2)
	[12]: 'VYUY' (VYUY 4:2:2)
	[13]: 'NC12' (Y/CbCr 4:2:0 (128b cols))

It also seems to be trying to pass mjpeg directly to the H264 encoder, and that certainly isn't supported.

You may be able to do a similar -pix_fmt yuv420p to force swscaler to convert the image data, or you may be able to use "yuyv422" to reduce the amount of computation required to do the conversion.

[MaoJianwei]

@6by9 That's amazing! It works, thank you!

-pix_fmt yuv420p
-vcodec h264_v4l2m2m

ffmpeg -y -s 640x480 -i /dev/video0 -pix_fmt yuv420p -vf "drawtext='fontsize=16:fontfile=FreeSerif.ttf:text=%{localtime\:%Y-%m-%d %H-%M-%S}:fontcolor=white'" -vcodec h264_v4l2m2m -an -f flv -y "rtmp://xxx.xxx.com:0000/hls/xxx"

ffmpeg -y -s 640x480 -i /dev/video0 -stream_loop -1 -re -i "/home/xxx/xiaowangge.wav" -vf "drawtext='fontsize=16:fontfile=FreeSerif.ttf:text=%{localtime\:xxx %Y-%m-%d %H-%M-%S}:fontcolor=white'" -pix_fmt yuv420p -vcodec h264_v4l2m2m -c:a aac -ar 44100 -b:a 320000 -f flv -y "rtmp://xxx.xxx.com:0000/hls/xxx"

[video4linux2,v4l2 @ 0x103cd30] Time per frame unknown
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 1722540.697327, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 640x480, 1000k tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[Parsed_drawtext_0 @ 0x10490d0] Using "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"
[swscaler @ 0x10613a0] deprecated pixel format used, make sure you did set range correctly
[h264_v4l2m2m @ 0x1041620]  <<< v4l2_encode_init: fmt=0/-1
[h264_v4l2m2m @ 0x1041620] Using device /dev/video11
[h264_v4l2m2m @ 0x1041620] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x1041620] requesting formats: output=YU12 capture=H264
Output #0, flv, to 'rtmp://xxx.xxx.com:0000/hls/xxx':
  Metadata:
    encoder         : Lavf58.45.100
    Stream #0:0: Video: h264 (h264_v4l2m2m) ([7][0][0][0] / 0x0007), yuv420p, 640x480, q=-1--1, 200 kb/s, 1000k fps, 1k tbn, 1000k tbc
    Metadata:
      encoder         : Lavc58.91.100 h264_v4l2m2m
frame=   60 fps= 13 q=-0.0 size=      50kB time=00:00:06.17 bitrate=  66.5kbits/s speed= 1.3x

Cheers~
Mao

[Peterkal2112]

I’m running into a CPU bottleneck when converting MJPEG to YUV420p for hardware-accelerated H.264 encoding using h264_v4l2m2m. The issue is that h264_v4l2m2m only accepts YUV420p input, but the conversion from MJPEG to YUV420p is CPU-bound, even though the actual H.264 encoding is handled by the GPU. This creates a significant performance hit, especially on devices with limited CPU resources like the Raspberry Pi.

Is there any way to offload the MJPEG to YUV420p conversion to the GPU, or another method to bypass this CPU bottleneck? Any suggestions for optimizing this workflow would be appreciated.

[6by9]

Closing as the original issue is fixed.

@Peterkal2112 your comment is unrelated. Please open a new thread on the forums as you are asking for assistance rather than reporting a bug - https://forums.raspberrypi.com/viewforum.php?f=38