Using h264_cuvid as hardware decoder

[digger18]

Hi,
I am trying to use h264_cuvid decoder to utilize NVIDIA GPU in order to decode the stream using PyAV.
My PyAV version is v6.0.0.

The code works, but it doesn't really utilize the h264_cuvid decoder, it still uses the CPU as decoder.

These are the relevant parts of my code, please let me know what am I doing wrong:

import av
from av.codec import CodecContext
from av.video import VideoFrame
from av.packet import Packet

=== SOME CODE GOES HERE ===

        try:
            cc = CodecContext.create('h264_cuvid', 'r')
            video = av.open(self.fifo_file_path)
        except Exception:
            logging.exception('EXCEPTION: FrameStore Error: Failed to open FIFO file')
            raise StopIteration

        stream = next(s for s in video.streams if s.type == 'video')

        end_pts = None
        if self.times is not None and self.local_play:
            start_pts = int(float(self.times[0]) / float(stream.time_base))
            stream.seek(start_pts)
            end_pts = int(float(self.times[1]) / stream.time_base)

        try:
            for packet in video.demux(stream):
                self.sequential_packet_error_counter = 0
                pts = packet.pts
                try:
                    for frame in cc.decode(packet):
                        self.decode_packet_error_counter = 0
                        if frame.pts is not None:
                            pts = frame.pts
                        else:
                            logging.warning('No frame pts received')
                        if self.local_play and self.times is not None and start_pts >= pts:
                            # logging.debug('Skipping frame with pts: {}, starting from pts: {}'.format(pts, start_pts))
                            continue
                        img = frame.to_nd_array(format='rgb24')
                        yield pts, packet.stream.time_base, img, self.start_time
                        # check if in turbo mode and exit after reach end pts
                        if self.local_play and self.times is not None and end_pts and int(frame.pts) > int(end_pts):
                            logging.debug('End of segment reached: {}'.format(end_pts))
                            raise StopIteration
                except Exception as e:
                    if isinstance(e, StopIteration):
                        raise StopIteration
                    self.decode_packet_error_counter += 1
                    if self.decode_packet_error_counter > self.max_sequential_packet_error:
                        logging.error('EXCEPTION: max_decode_packet_error reached: {}'.format(e))
                        raise StopIteration

        except Exception as e:
            # if we got stop iteration,
            if isinstance(e, StopIteration):
                raise
            logging.error('EXCEPTION: Failed to demux stream: {}'.format(e))
            self.sequential_packet_error_counter += 1
            if self.sequential_packet_error_counter > self.max_sequential_packet_error:
                logging.error('max_sequential_packet_error reached')
                raise StopIteration

=== CODE CONTINUES ===

FFMpeg build:

# ffmpeg --help   
ffmpeg version 4.0 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.9) 20160609
  configuration: --prefix=/usr/local --pkg-config-flags=--static --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --enable-pthreads --enable-shared --enable-libfreetype --enable-libx264 --enable-nonfree --enable-gpl --enable-openssl --enable-cuda --enable-cuvid --enable-nvenc --enable-libnpp --enable-cuda-sdk --disable-doc --disable-static
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100

Thanks in advance!

[inism]

Make sure that you use the latest nVidia Graphics Driver.
FFmpeg won't interface with nVidia Video Codec SDK for the driver versions older than 396.24.
https://stackoverflow.com/questions/51833519/compiling-ffmpeg-with-nvidia-cuda-failed-loading-nvcuvid

Also, in many cases, h264 decoders require extradata to split a packet (Access Unit, more precisely) into Network Abstract Layer Units (NALUs) properly, and you can copy the data from the codec_context of target video stream.
https://stackoverflow.com/questions/24884827/possible-locations-for-sequence-picture-parameter-sets-for-h-264-stream/24890903#24890903
Issue #155

Below is a rough example of decoding an h264 video stream with the 'h264_cuvid' decoder.

import av

video = av.open(VIDEO_FILE_PATH)
target_stream = video.streams.video[0]

ctx = av.Codec('h264_cuvid', 'r').create()
ctx.extradata = target_stream.codec_context.extradata

for packet in video.demux(target_stream):
    for frame in ctx.decode(packet):
        print(frame)

ctx.extradata = b''

There's a bug (as of PyAV 6.2.0) in the setter of extradata property of CodecContext class, which raises an AttributeError.
It can be fixed by modifying Line 99 of av/codec/context.pyx
from self.ptr.extradata_size = self.extradata_source.size
to self.ptr.extradata_size = self.extradata_source.length

Also, for my case, deleting (no matter explicitly deleted or implicitly done by the garbage collector) an instance of CodecContext whose extradata is manually set caused the script to crash.
This was avoided by re-assigning an empty bytes to extradata at the end.

[junedgar]

@inism how can I install av to support hardware acceleration?
I did these operations, but i still get error when i used your scripts.

1.install the ffmpeg with cuda support.

• git clone ffmpeg

• ./configure --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp --extra-cflags=-I/usr/local/cuda-10.0/include --extra-ldflags=-L/usr/local/cuda-10.0/lib64 --enable-shared

• make -j8

• make install

2. install av from source

• git clone -b hwaccel https://github.com/rvillalba-novetta/PyAV.git

• source scripts/activate

• make

3.Then I followed your script, get error blew:
'>>> import av

ctx = av.Codec('h264_cuvid', 'r').create()
Traceback (most recent call last):
File "", line 1, in
File "av/codec/codec.pyx", line 67, in av.codec.codec.Codec.cinit
self._init(name)
File "av/codec/codec.pyx", line 76, in av.codec.codec.Codec._init
raise UnknownCodecError(name) av.codec.codec.UnknownCodecError: h264_cuvid
'

Can you give me some advices?

[inism]

@junedgar

It looks like you did something wrong when you build PyAv,
but I'm afraid I'm not able to provide correct advice because I have been using PyAv on Windows only.

On Windows, when you build av, a path to ffmpeg's .h and .lib files should be given to allow PyAv can properly interface to the specific ffmpeg version you want to use, and it can be done like this:
python.exe setup.py build --ffmpeg-dir="PATH_TO_FFMPEG_WIN_DEV_DISTRIBUTION",
where PATH_TO_FFEMPG_WIN_DEV_DISTRIBUTION is the path to the official distribution of ffmpeg for windows development.

ADDED =============

I realized that you have installed rvillalba-novetta's branch, not the official one.
I suggest you to try the latest official release.
Support for hardware acceleration is one of the works made recently (from 6.1?),
so you might fail it with older releases or their folks.

[junedgar]

@inism
It's very kind of you!
I builded the av from source with version 6.2, but i still have this problem.
May something in ubuntu system.
Thank you for your reply!

[jlaine]

@mikeboers does this fall under the category of "won't fix" bugs related to hardware acceleration?

[mikeboers]

@jlaine Yeah... I’m not sure what to do here. Seems like an end build problem or an ffmpeg problem, not PyAV.