FFmpeg Libraries: Exact Constant Segment Duration for HLS

Question

We use the FFmpeg git -ee94362 libavformat v55.2.100 libraries. Our goal is to combine two streams (video and audio) into an M3U8 playlist using HLS. In addition, we want the duration of each TS segment segment to be exactly 3.0 s (frame rate 25 frames per second).

To achieve this, we try to set several parameters and properties, namely: - segment_time
- keyint_min - scenechange_threshold - gop_size - force_key_frames.

And our code is as follows:

AVCodecContext *codec_ctx = NULL; AVFormatContext *ofmt_ctx = NULL; int ret = 0, gopSize = (int)(3.0 * 25); // 3 sec * 25 fps // ofmt_ctx and codec_ctx initialization and filling are OK, but: codec_ctx->time_base.num = 1; codec_ctx->time_base.den = 25 // fps // It seems, that the following three lines have no effect without explisit setting of the "hls_time" property codec_ctx->keyint_min = gopSize; // in FFMpeg application, the corresponding option is "-keyint_min 3" codec_ctx->scenechange_threshold = 0; // in FFMpeg application, the corresponding option is "-sc_threshold 0" codec_ctx->gop_size = gopSize; // in FFMpeg application, the corresponding option is "-g 3" ret = av_opt_set_double(ofmt_ctx, "hls_time", 3.0, AV_OPT_SEARCH_CHILDREN); // Any of the following lines causes "Option not found" error. ret = av_opt_set(codec_ctx->priv_data, "profile", "main", AV_OPT_SEARCH_CHILDREN); ret = av_opt_set(codec_ctx->priv_data, "preset", "ultrafast", AV_OPT_SEARCH_CHILDREN); ret = av_opt_get(ofmt_ctx, "segment_time", AV_OPT_SEARCH_CHILDREN, &str); ret = av_opt_set((ofmt_ctx, "segment_time", "3.0", AV_OPT_SEARCH_CHILDREN); 

In any case, the duration of the TS files is different, (~ 2-3 seconds), and not EXACTLY 3.0 seconds. Our question: What is the best way to solve the problem?

Andrey Mochenov.

Answer 1

The main problem you are facing is probably that your video file does not have key frames in the fit positions . This is especially a problem if you just copy the streams from the input.

FFmpeg depends on key frames to calculate when to "cut" a segment. It makes sense when you think about it. You cannot just make a cut between two keyframes, since each segment must be fully functional in itself. Now it can be argued that FFmpeg should just insert new keyframes on its own, but that would be too friendly, right?)

Fortunately, you can pin keyframes with FFmpeg. Either use the parameter, or set the flag in the code yourself. You said you already tried to force keyframes, but I assume that you did not do it right.

This my test gives good results. This is just a command line, sorry, but you already know how to apply command line parameters in your code, so everything should be fine. Also note that I do not use the "hls_XXX" parameters, because: a) I honestly do not trust them, and b) in this way, I suppose, it should also work for non-HLS streams.

 ffmpeg -i inputFile.mov -force_key_frames "expr:gte(t,n_forced*10)" -strict -2 -c:a aac -c:v libx264 -f segment -segment_list_type m3u8 -segment_list_size 0 -segment_time 10.0 -segment_time_delta 0.1 -segment_list stream/test.m3u8 stream/test%02d.ts 

You can see exactly how the force_key_frames command works here .

So far, I have implemented the above C ++ command with some additions. But without "force_key_frames", when I set keyframes manually during the transcoding process. Here is what I did:

 AVDictionary* headerOptions(0); av_dict_set(&headerOptions, "segment_format", "mpegts", 0); av_dict_set(&headerOptions, "segment_list_type", "m3u8", 0); av_dict_set(&headerOptions, "segment_list", _playlistFileName.c_str(), 0); av_dict_set_int(&headerOptions, "segment_list_size", 0, 0); av_dict_set(&headerOptions, "segment_time_delta", TO_STRING(1.00).c_str(), 0); av_dict_set(&headerOptions, "segment_time", TO_STRING(_segmentDuration).c_str(), 0); av_dict_set_int(&headerOptions, "reference_stream", _videoStream->index, 0); av_dict_set(&headerOptions, "segment_list_flags", "cache+live", 0); avformat_write_header(_formatContext, &headerOptions); 

And here is the result of m3u8:

 #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-ALLOW-CACHE:YES #EXT-X-TARGETDURATION:11 #EXTINF:10.083333, test00.ts #EXTINF:10.000000, test01.ts #EXTINF:10.000000, test02.ts #EXTINF:10.000000, test03.ts #EXTINF:10.000000, test04.ts #EXTINF:10.000000, test05.ts #EXTINF:0.083333, test06.ts #EXT-X-ENDLIST 

This is not ideal (the first part is somewhat in some way), but I am convinced that you will not get better results than this.

Of course, the best option would be to make sure that your input files always have the correct key frames when copying streams simply, but sometimes you have no control over what files you get.

Side note

When you use FFmpeg in code, always try to do what you do in code using the cli ffmpeg command. If you can get it to work this way, you at least know what parameters to set in the code. And if it works with the command line tool, you know that this should be possible in the code in some way;)

Answer 2

You can also try to achieve a 3 second segment of (approximately) duration by changing ffmpeg. As @theSHEEP pointed out ffmpeg, I wait until I am torn before making a cut. You can change this ffmpeg behavior by forcing it to make a reduction in "your time" than wait for me to frame.

  ffmpeg/libavformat/segment.c, 795 static int seg_write_packet(AVFormatContext *s, AVPacket *pkt) 835 if (pkt->stream_index == seg->reference_stream_index && 836 pkt->flags & AV_PKT_FLAG_KEY && 837 seg->segment_frame_count > 0 && 838 (seg->cut_pending || seg->frame_count >= start_frame || 839 (pkt->pts != AV_NOPTS_VALUE && 840 av_compare_ts(pkt->pts, st->time_base, 841 end_pts-seg->time_delta, AV_TIME_BASE_Q) >= 0))) 

I would change the line 835 to 841 to fit my requirements. (Comment line number 836 and try and remember that FFMPEG is LGPL)

The HLS IETF version recommends:

The server MUST try to split the source media at points that support efficient decoding of individual media segments, for example. by package and key frame boundaries

I will read it as a recommendation than a requirement .;)

Answer 3

It is not very good to make a place that does not have an I-frame be cut, because if you want to decode only frames in this particular segment, they will be filled with gray rectangles. There is simply not enough data to correctly decode the full frame.

It would be best to encode the sequence first with

 AVCodecContext *enc_ctx; ... av_opt_set_int(enc_ctx, "sc_threshold", sc_threshold, 0); enc_ctx->gop_size = 3 * 25; av_opt_set_int(enc_ctx, "keyint_min", min_keyint, 0); 

Later, as soon as your encoding is completed, you can select a file for HLS separately or do it during encoding. In my particular use case, I did this after the entire coding cycle was completed. The code from @TheSHEEEP helped with this, but the options he used were not the ones I needed.

 size_t f = output_filename.find_last_of("."); string ofn = output_filename.substr(0, f); ofn.append(".m3u8"); avformat_alloc_output_context2(&ofmt_ctx, NULL, "hls", ofn.c_str()); AVDictionary* headerOptions = NULL; av_dict_set(&headerOptions, "hls_segment_type", "mpegts", 0); av_dict_set(&headerOptions, "hls_playlist_type", "event", 0); av_dict_set_int(&headerOptions, "hls_list_size", 0, 0); av_dict_set(&headerOptions, "segment_time_delta", "1.0", 0); av_dict_set(&headerOptions, "hls_flags", "append_list", 0); ret = avformat_write_header(ofmt_ctx, &headerOptions); 

Where ofmt_ctx is outputted by AVFormatContext. The output file is the same as in the message from @TheSHEEEP.