How do I split a rtsp stream in multiple files with remux?

[linchiyu]

Overview

I'm trying to split a rtsp stream in multiple files. I was successfull doing it but the files that came out have a time length issue, it increments the last file length on his own. I'm doing 10 seconds cuts for each file, the first file gets 10 seconds but the second file gets 20 seconds time length with 10 seconds of content, the third get 30 seconds file with last 10 seconds of data only.

Here is my code:

import time

from os import path

import av

inputContainer = av.open("rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4", "r")

while True:

    dateString = time.time()

    start = time.time()

    videoFilePath = f"output{dateString}.avi"

    in_stream = inputContainer.streams.video[0]

    output_time = 0

    with av.open(f"{videoFilePath}", "w") as output:

        out_stream = output.add_stream(template=in_stream)

        for packet in inputContainer.demux(in_stream):

            if packet.dts is None:

                continue

            packet.stream = out_stream

            packet.pts = packet.pts - output_time

            packet.dts = packet.dts - output_time

            output.mux(packet)

            cur_time = time.time() - start

            if cur_time >= 10:

                output_time = packet.dts

                break

Expected behavior

Each file produced should be 10 seconds on metadata.

Actual behavior

The files gets larger time length, if you open the file it shows no image up until the last 10 seconds of the file.

Investigation

I've tried to change pts and dts timers but it changes nothing.
I also tried to look for changing the metadada in container or the stream, but found nothing

[e-d-n-a]

I was trying to find a solution in various ways, but something is seriously wrong with your example input stream!
(rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4)

There are several ways to approach the problem and I tested them using the stream (live) or the downloaded video (as mp4).
The first thing would be to test splitting the video (as a stream or from file) into fragments using the "ffmpeg" CLI tool.

Option 1 (single fragment, maybe also possible using PyAV):

ffmpeg -ss 00:00:10 -i "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4" -t 10 -c copy -avoid_negative_ts 1 part_10.mp4

Sources:
https://trac.ffmpeg.org/wiki/Seeking#Cuttingsmallsections (seeking before -i is supposed to reset the timestamps in the output)
for more info on "avoid_negative_ts" see:
https://ffmpeg.org/ffmpeg-all.html#toc-Format-Options
https://github.com/FFmpeg/FFmpeg/blob/7e0d640edf6c3eee1816b105c2f7498c4f948e74/libavformat/avformat.h#L1579
A bash shell script for splitting videos

Problems:

• Output file(s) still has/have same issue (not starting at 0s and showing greater length than 10s)
• "avoid_negative_ts" parameter seems to not work (neither with option set to 1="make_non_negative", nor 2="make_zero")
• "ffmpeg" shows several issues with timestamps in log:

Input #0, rtsp, from 'rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4':
  Metadata:
    title           : BigBuckBunny_115k.mp4
  Duration: 00:10:34.63, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 12000 Hz, stereo, fltp
    Stream #0:1: Video: h264 (High), yuv420p(progressive), 240x160 [SAR 32:27 DAR 16:9], 24 fps, 24.
08 tbr, 90k tbn, 48 tbc
Output #0, mp4, to 'part_10.mp4':
  Metadata:
    title           : BigBuckBunny_115k.mp4
    encoder         : Lavf58.45.100
    Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 240x160 [SAR 32:27 DA
R 16:9], q=2-31, 24 fps, 24.08 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 12000 Hz, stereo, fltp
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0000000003b82ec0] Timestamps are unset in a packet for stream 0. This is deprecated and will
stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0000000003b82ec0] Invalid DTS: -3750 PTS: -10530 in output stream 0:0, replacing by guess
[mp4 @ 0000000003b82ec0] pts has no value
[mp4 @ 0000000003b82ec0] Non-monotonous DTS in output stream 0:0; previous: 0, current: -10530; chan
ging to 1. This may result in incorrect timestamps in the output file.
[mp4 @ 0000000003b82ec0] Non-monotonous DTS in output stream 0:0; previous: 1, current: -6750; chang
ing to 2. This may result in incorrect timestamps in the output file.
[mp4 @ 0000000003b82ec0] Non-monotonous DTS in output stream 0:0; previous: 2, current: -2970; chang
ing to 3. This may result in incorrect timestamps in the output file.
frame=  246 fps= 24 q=-1.0 Lsize=     204kB time=00:00:09.96 bitrate= 167.4kbits/s speed=0.959x
video:157kB audio:40kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.561142%

Option 2 (segment muxer, here using downloaded video):

ffmpeg -i "full.mp4" -t 62 -c copy -f segment -segment_time 10 -segment_time_delta 0.6 -reset_timestamps 1 "out3_%02d.mp4"
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'full.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : BigBuckBunny_115k.mp4
    encoder         : Lavf58.45.100
  Duration: 00:10:34.62, start: 0.000000, bitrate: 120 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 240x160 [SAR 32:27 DAR 16:9],
 83 kb/s, 24 fps, 24 tbr, 90k tbn, 48 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 12000 Hz, stereo, fltp, 32 kb/s (default)

    Metadata:
      handler_name    : SoundHandler
[segment @ 0000000000444300] Opening 'out3_00.mp4' for writing
Output #0, segment, to 'out3_%02d.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : BigBuckBunny_115k.mp4
    encoder         : Lavf58.45.100
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 240x160 [SAR 32:27 DAR 16:9],
 q=2-31, 83 kb/s, 24 fps, 24 tbr, 12288 tbn, 24 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 12000 Hz, stereo, fltp, 32 kb/s (default)

    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[segment @ 0000000000444300] Non-monotonous DTS in output stream 0:0; previous: 512, current: 512; c
hanging to 513. This may result in incorrect timestamps in the output file.
[segment @ 0000000000444300] Non-monotonous DTS in output stream 0:0; previous: 513, current: 512; c
hanging to 514. This may result in incorrect timestamps in the output file.
[segment @ 0000000000444300] Opening 'out3_01.mp4' for writing
[segment @ 0000000000444300] Opening 'out3_02.mp4' for writing
[segment @ 0000000000444300] Opening 'out3_03.mp4' for writing
[segment @ 0000000000444300] Opening 'out3_04.mp4' for writing
[segment @ 0000000000444300] Opening 'out3_05.mp4' for writing
[segment @ 0000000000444300] Opening 'out3_06.mp4' for writing
frame= 1491 fps=0.0 q=-1.0 Lsize=N/A time=00:01:01.99 bitrate=N/A speed= 930x
video:689kB audio:244kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

(creating first 6 fragments from downloaded file)

Sources:
https://askubuntu.com/questions/948304/how-to-automatically-segment-video-using-ffmpeg-without-re-encoding
https://ffmpeg.org/ffmpeg-formats.html#segment_002c-stream_005fsegment_002c-ssegment

Problems:

• Not possible using PyAV!?
• Had some issues, when reencoding/remuxing or needs more detailed settings (stream copy worked fine and precisely though!)

Option 3 (reassigning timestamps via raw h264 stream, somewhat possible with PyAV):

Sources:
https://superuser.com/questions/1159056/ffmpeg-recreate-timestamps-without-reencoding
https://pyav.org/docs/9.0.2/cookbook/basics.html#parsing

Problems:

• PyAV cannot create raw stream (has to be done using CLI tools in subprocess -> see PyAV docu example in sources)
• needs complex manual (re-)setting of all timestamps!?

Conclusion:

• Example video already has some timestamp issues
• Example video, despite being delivered as a stream, makes heavy use of B-frames
• Example video has I-frames every 1s (makes exact splitting actually possible -> see option 2)
• Splitting should be done on I-frames (keyframe)
• Uncertain whether format/container/muxer/stream options like "avoid_negative_ts" can be used to reset timestamps
• Test program using option 3 has yet to be created and explored

Tools used:

• Python v3.7.4 (32-bit, Windows)
• PyAV v8.02
• ffmpeg/ffprobe v4.3.1-2020-11-19-full_build-www.gyan.dev

Hints

• I only tried outputting to a MOV/mp4 container
  (one could experiment with avi, mkv or better TS as well)
• Checking picture types in video

ffprobe -i full.mp4 -select_streams v:0 -show_entries frame=pkt_pts,pict_type

• Checking video for I-frames

ffprobe -v error -skip_frame nokey -show_entries frame=pkt_pts_time -select_streams v -of csv=p=0 full.mp4

• Other ways to reset PTS or splitting into TS segments
  https://stackoverflow.com/questions/44425611/how-to-reset-pts-for-stream
  https://trac.ffmpeg.org/wiki/Concatenate#samecodec
  https://hlsbook.net/segmenting-video-with-ffmpeg/

[e-d-n-a]

I explored option 1 from my previous post also using PyAV (v8,0,2) with a modified version of your script:

import time

from os import path

import av

inputContainer = av.open("rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4", "r")
in_stream = inputContainer.streams.video[0]

av_options = {'avoid_negative_ts': '-1'}
packet = None
frag_offset = 0.0 # secs
frag_length = 10.0 # secs
frag_dts = 0 # 1st original reference timestamp of a fragment

while True:

	videoFilePath = f"output_{frag_offset:.0f}.mp4"

	with av.open(f"{videoFilePath}", "w") as outputContainer:

		out_stream = outputContainer.add_stream(template=in_stream)
		# print(outputContainer.container_options)
		outputContainer.container_options.update(av_options)
		
		# output keyframe from last split as first frame of new fragment
		if packet:
			packet.stream = out_stream
			outputContainer.mux(packet)

		for packet in inputContainer.demux(video=0):

			if packet.dts is None:
				# packet.stream = out_stream
				# outputContainer.mux(packet)
				continue
			
			cur_time = float(packet.dts * packet.time_base)
			
			orig_dts = packet.dts
			packet.dts = packet.dts - frag_dts
			packet.pts = packet.pts - frag_dts
			print(frag_offset, orig_dts, packet.dts, packet.pts, packet.time_base, cur_time)
			
			if cur_time >= frag_offset+frag_length and packet.is_keyframe:
				frag_offset = cur_time
				frag_dts = orig_dts
				break
			else:
				packet.stream = out_stream
				outputContainer.mux(packet)

Well, in this later stage I also tried to manipulate the timestamps of each packet, but I originally was expecting this to be not needed. It finally fails as there are probably weird timestamp issues within the example file, that one has to deal with and fix somehow.

I mostly was interested in whether PyAV allows you setting the muxer option "avoid_negative_ts" to maybe force ffmpeg to reset the timestamps in each output fragment!
Yet, I couldn't confirm, that applying it in this way (and setting it to value of 1 or 2) has really any effect.

Looking at the ffmpeg documentation it definitely seems to be an option on the muxer/container though:

19 Format Options

The libavformat library provides some generic global options, which can be set on all the muxers and demuxers. In addition each muxer or demuxer may support so-called private options, which are specific for that component.

Options may be set by specifying -option value in the FFmpeg tools, or by setting the value explicitly in the AVFormatContext options or using the libavutil/opt.h API for programmatic use.

As I couldn't make it work even using the ffmpeg CLI tool, I stopped trying.

References:
https://ffmpeg.org/ffmpeg-all.html#toc-Format-Options
https://pyav.org/docs/8.0.1/api/container.html#av.container.Container.container_options
https://pyav.org/docs/8.0.1/api/stream.html
https://pyav.org/docs/8.0.1/api/packet.html
https://pyav.org/docs/8.0.1/api/time.html

[e-d-n-a]

Fixed (working now!)

Sorry my posted modification of your script still had a small bug, that produced a "non-monotonic error" as the initial packet for the next fragment had the previous offset applied.
I now fixed it and it works as expected producing accurate 10s-fragments, that can be played just fine.

import time

from os import path

import av

#inputContainer = av.open("rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4", "r")
inputContainer = av.open("full.mp4", "r")
in_stream = inputContainer.streams.video[0]

av_options = {'avoid_negative_ts': '1'}
packet = None
frag_offset = 0.0 # secs
frag_length = 10.0 # secs
frag_pts = 0

while True:

	videoFilePath = f"out4_{frag_offset:.0f}.mp4"

	with av.open(f"{videoFilePath}", "w") as outputContainer:

		out_stream = outputContainer.add_stream(template=in_stream)
		print('stream_cc:',out_stream.codec_context.time_base)
		# print(outputContainer.flags)
		# break
		outputContainer.container_options.update(av_options)
		
		if packet:
			packet.dts = packet.dts - frag_pts
			packet.pts = packet.pts - frag_pts
			packet.stream = out_stream
			print('init :',packet,frag_offset)
			outputContainer.mux(packet)

		for packet in inputContainer.demux(out_stream):

			if packet.dts is None:
				continue
			
			cur_time = float(packet.pts * packet.time_base)
			
			orig_pts = packet.pts
			print('cont:',orig_pts, packet.time_base, cur_time)
			
			if cur_time >= frag_offset+frag_length and packet.is_keyframe:
				frag_offset = cur_time
				frag_pts = orig_pts
				break
			else:
				packet.dts = packet.dts - frag_pts
				packet.pts = packet.pts - frag_pts
				packet.stream = out_stream
				outputContainer.mux(packet)

I also took the time to look at the internals of ffmpeg and PyAV to check what is going on while muxing using the segment muxer with ffmpeg or the OutputContainer with PyAV.
The segment muxer modifies the timestamps of the packets based on packet pts, so I did it similarly.
Both use also the flag "-autobsf" on the container, which I checked (see sources).
The muxing in OutputContainer includes rescaling the timestamps to the container time_base using "lib.av_packet_rescale_ts()", so I checked that there was a valid value set from the stream template.

References:
computation of new timestamps in segment muxer of ffmpeg
timestamp rebasing in output muxing of PyAV
rescaling function used in packet rebasing function of PyAV
example fragment produced by corrected script

[linchiyu]

Using following ffmpeg command and it seems to run smoothly
ffmpeg -i rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4 -y -c:v copy -an -strftime 1 -reset_timestamps 1 -segment_time 10 -f segment output-%Y%m%d%H%M%S.mp4

Your solution is good but still it breaks if we use the live stream and not on the downloaded file.

In the real project i'm using live feed IP Cameras URLs that have the same problem.

[linchiyu]

It might be a stream problem and we can't work around it.