Allow nvenc encoders

[tbenst]

Overview

NVENC is more than twice as fast as cpu encoding at similar or higher quality.

Existing FFmpeg API

https://ffmpeg.org/doxygen/trunk/structNvencContext.html

Expected PyAV API

PyAV should support h264_nvenc and nvenc_hevc for codec options

[tbenst]

I saw on gitter it being said that this is already supported. Turns out, my ffmpeg installation has an issue:

> ffmpeg -i f2_e1_FishVR.mp4 -c:v h264_nvenc nvenc_test.mp4
[h264_nvenc @ 0x5e28c0] Cannot load libcuda.so.1
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

Closing for now as may not be a PyAV issue

[pwuertz]

My ffmpeg command works fine using -c:v h264_nvenc as video encoder, but in PyAV the codec isn't recognized:

>>> codec = av.Codec("h264_nvenc", "w")
UnknownCodecError: h264_nvenc 

How to select hardware encoders like h264_nvenc or h264_v4l2m2m in PyAV?

[pwuertz]

All right, I think the problem was the PyAV build I got from Ubuntu 20.10. When I tried to use the nvenc codecs i had the apt python3-av package installed, which apparently lacks a lot of codec options compared to the system's ffmpeg build.

When building PyAV from source, all those codec options are in fact available:

assert "h264_nvenc" in av.codecs_available
assert "hevc_nvenc" in av.codecs_available
# all good

Here is a small, timed, low-level encoder based on PyAV:

def encode_frames(
    frames: typing.Iterable[av.VideoFrame], codec_name: str
) -> typing.Sequence[av.Packet]:
    """Encode a sequence of video frames using given codec_name."""

    frame_size = frames[0].width, frames[0].height
    codec = av.Codec(codec_name, "w")
    codec_ctx = codec.create()
    try:
        codec_ctx.pix_fmt = "yuv420p"
        codec_ctx.width, codec_ctx.height = frame_size
        codec_ctx.framerate = 20

        packets = []
        packets += codec_ctx.encode(frames[0])
        t1 = time.time()
        for frame in frames[1:]:
            packets += codec_ctx.encode(frame)
        packets += codec_ctx.encode()
        t2 = time.time()
    finally:
        if codec_ctx.is_open:
            codec_ctx.close()

    fps = (len(frames) - 1) / (t2-t1)
    print(f"Encoded at {fps:.1f} fps ({codec_name})")
    return packets

Which gives me very nice, GPU accelerated results on a desktop (100 frames, 800x600):

Encoded at 153.9 fps (h264)
Encoded at 1610.7 fps (h264_nvenc)

[tbenst]

Woah that’s an awesome speed-up!! I only ever saw a 2x speed up with nvenc. May I ask what CPU / GPU you benchmarked this on? I tested with i9-10980XE and Titan RTX. Or maybe difference is in code/settings

P.S. sounds like okay to close issue?

[pwuertz]

I think it's okay to close, the encoded video frames look fine too.

The speedup is probably due to my old CPU (i7-4790K) being pitted against the NVENC from the RTX 2070. Also, the timing method I used in my example above excludes the time to create and initialize the Codec by not taking the first frame into account. This might otherwise reduce the measured speedup when using short examples.

[lamhoangtung]

Hi @pwuertz @tbenst, I'm trying to use the nvenc codecs as you did, the ffmpeg binary work great for me on something like: ffmpeg -i input.mp4 -c:v h264_nvenc -preset hq output.mp4 but pyav failed with:

Traceback (most recent call last):
  File "test_enc.py", line 51, in <module>
    codec = av.CodecContext.create('h264_nvenc', 'r')
  File "av/codec/context.pyx", line 67, in av.codec.context.CodecContext.create
  File "av/codec/codec.pyx", line 70, in av.codec.codec.Codec.__cinit__
  File "av/codec/codec.pyx", line 79, in av.codec.codec.Codec._init
av.codec.codec.UnknownCodecError: h264_nvenc

I built ffmpeg separately and then install the hwaccel branch (90e334f) of pyav with pip like this: pip install git+git://github.com/PyAV-Org/PyAV.git@90e334f1b0e22a00c6edf05b51f1ec8da56315f1

Since you had done this before and not having this problem. Can you point me some where here ? Thanks

Also, here is my ffmpeg build config:

ffmpeg version 4.4.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-avresample --enable-cuda --enable-cuvid --enable-fontconfig --enable-gpl --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libfdk_aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libnpp --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libsrt --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzmq --enable-nonfree --enable-nvenc --enable-openssl --enable-postproc --enable-shared --enable-small --enable-version3 --extra-cflags='-I/opt/ffmpeg/include -I/opt/ffmpeg/include/ffnvcodec -I/usr/local/cuda/include/' --extra-ldflags='-L/opt/ffmpeg/lib -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib32/' --extra-libs=-ldl --extra-libs=-lpthread --prefix=/opt/ffmpeg
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100

[pwuertz]

It looks like you're trying to create an "h264_nvenc" encoder for decoding ("r").

[lamhoangtung]

Thanks a lot @pwuertz.

[vade]

FWIW, I just tested this via a pretty straightforward test setup using docker.

1. docker pull jrottenberg/ffmpeg:4.4.2-nvidia2004
2. docker run --gpus all -v ~/videos:/videos -it --entrypoint /bin/bash --rm jrottenberg/ffmpeg:4.4.2-nvidia2004 - this mounts a local test folder of ~/videos to a /video in the container. Throw a test video there.
3. The above docker doesn't have pyAV or python so install python and pyAV dependencies:
4. apt-get install python
5. apt-get install pip
6. apt-get install pkg-config
7. pip install av --no-binary av

Then you can run the following with h264_cuvid and test vs h264:

import av
from datetime import datetime

video_path = "some_file.mp4"


video = av.open(video_path)
target_stream = video.streams.video[0]

print(target_stream)

ctx = av.Codec('h264', 'r').create()
#ctx = av.Codec('h264_cuvid', 'r').create()
ctx.extradata = target_stream.codec_context.extradata

start_time = datetime.now()

for packet in video.demux(target_stream):
    for frame in ctx.decode(packet):
        print(frame)

end_time = datetime.now()

total_time = end_time - start_time

print("Took", total_time) 

I get about a 6x diff on a 3090:

h264: Took 0:00:18.961413
h264_cuvid: 0:00:02.938973