Hang inside "VCHIQ completion" while transcoding MPEG2 to H264 using GPU

[minfrin]

While trying to transcode MPEG2 to H264 video using the GPU via gstreamer, the pipeline works for a period of time (usually minutes, sometimes hours), then transcoding stops suddenly and hangs.

Connecting to the gst-launch process with gdb shows all threads polling except for this thread:

19 Thread 0x75efa470 (LWP 2489) "VCHIQ completio" 0x76b91dcc in ioctl ()
from /lib/arm-linux-gnueabihf/libc.so.6

The backtrace of the thread is below.

Is there such a thing as a debug version of the firmware with optimisation disabled and symbols present so I can debug this?

(gdb) thread 19
[Switching to thread 19 (Thread 0x75efa470 (LWP 2489))]
#0 0x76b91dcc in ioctl () from /lib/arm-linux-gnueabihf/libc.so.6

(gdb) bt
#0 0x76b91dcc in ioctl () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0x76db2030 in completion_thread () from /opt/vc/lib/libvchiq_arm.so
#2 0x760f2da0 in vcos_thread_entry () from /opt/vc/lib/libvcos.so
#3 0x76c04c00 in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
#4 0x76b99728 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
#5 0x76b99728 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

The gstreamer launch pipeline looks like this:

gst-launch-1.0 -v udpsrc uri=udp://[ff08::30]:5004 "caps=application/x-rtp,media=(string)video,clock-rate=(int)90000" ! rtpbin ! rtpmp2tdepay ! progressreport update-freq=5 ! tsparse ! tsdemux name=demux ! queue ! mpegvideoparse ! omxmpeg2videodec ! progressreport update-freq=5 ! queue ! videoscale ! video/x-raw,width=384,height=256 ! videorate ! omxh264enc target-bitrate=200000 control-rate=variable inline-header=true ! queue ! progressreport update-freq=5 ! mpegtsmux name=mux ! hlssink location=/var/www/stream/segment2%05d.ts playlist-location=/var/www/stream/output2.m3u8 playlist-root=http://192.168.225.3/stream/ target-duration=1 demux. ! queue ! mpegaudioparse ! mpg123audiodec ! audiorate ! voaacenc ! aacparse ! queue ! mux.

[pelwell]

Blocked in the kernel is the normal state for that thread; it is there to pick up asynchronous notifications and callbacks from the kernel.

[minfrin]

Is there a way to debug what is going on inside the GPU firmware?

[popcornmix]

You can't debug the GPU firmware.
You could add "start_debug=1" to config.txt and reboot to enable assert logging.
Then after the hang, run "sudo vcdbg log msg" which may have some complaints in the log from gpu.

[minfrin]

Running "sudo vcdbg log msg" before and then after the gst-launch failure, no extra lines are added to the log.

[6by9]

I suspect it should actually be "sudo vcdbg log assert" unless there have been changes in vcdbg or logging (I haven't checked recently)

[popcornmix]

@6by9 you are correct, the asserts don't appear in the msg log.

[Ruffio]

@minfrin do you still experience this issue or can it be closed?

[Ruffio]

@popcornmix please consider to close this as there has been no further information after your suggestion to use start_debug=1 15 Aug 2015