Using ffmpeg to capture frames from a webcam and audio from micro and save to file

Question

Over the past few weeks, I have been struggling with the ffmpeg API, as I cannot find clear documentation, and it is also difficult to search, since all the solutions that I find on the Internet do not include the API c, but the ffmpeg.c command line. I am creating a program that is supposed to capture video from a webcam and audio, show frames on the screen and record both audio and frames in a video file. I also use QT as the basis for this project.

I managed to show the frames on the screen and even record them, but my problem is recording both audio and video. I decided to create a simpler test program that only saves the stream to a file without showing frames on the screen, starting with the remuxing.c example in the ffmpeg documentation. My code is as follows:

//This is the variables on the .h AVOutputFormat *ofmt; AVFormatContext *ifmt_ctx, *ofmt_ctx; QString cDeviceName; QString aDeviceName; int audioStream, videoStream; bool done; //The .cpp #include "cameratest.h" #include <QtConcurrent/QtConcurrent> #include <QDebug> CameraTest::CameraTest(QString cDeviceName, QString aDeviceName, QObject *parent) : QObject(parent) { done = false; this->cDeviceName = cDeviceName; this->aDeviceName = aDeviceName; av_register_all(); avdevice_register_all(); } void CameraTest::toggleDone() { done = !done; } int CameraTest::init() { ofmt = NULL; ifmt_ctx = NULL; ofmt_ctx = NULL; QString fullDName = cDeviceName.prepend("video=") + ":" + aDeviceName.prepend("audio="); qDebug() << fullDName; AVInputFormat *fmt = av_find_input_format("dshow"); int ret, i; if (avformat_open_input(&ifmt_ctx, fullDName.toUtf8().data(), fmt, NULL) < 0) { fprintf(stderr, "Could not open input file '%s'", fullDName.toUtf8().data()); return -1; } if ((ret = avformat_find_stream_info(ifmt_ctx, 0)) < 0) { fprintf(stderr, "Failed to retrieve input stream information"); return -1; } av_dump_format(ifmt_ctx, 0, fullDName.toUtf8().data(), 0); avformat_alloc_output_context2(&ofmt_ctx, NULL, NULL, "test.avi"); if (!ofmt_ctx) { fprintf(stderr, "Could not create output context\n"); ret = AVERROR_UNKNOWN; return -1; } ofmt = ofmt_ctx->oformat; for (i = 0; i < ifmt_ctx->nb_streams; i++) { AVStream *in_stream = ifmt_ctx->streams[i]; AVStream *out_stream = avformat_new_stream(ofmt_ctx, in_stream->codec->codec); if (ifmt_ctx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) { videoStream = i; } else if (ifmt_ctx->streams[i]->codec->codec_type == AVMEDIA_TYPE_AUDIO) { audioStream = i; } if (!out_stream) { fprintf(stderr, "Failed allocating output stream\n"); ret = AVERROR_UNKNOWN; return -1; } ret = avcodec_copy_context(out_stream->codec, in_stream->codec); if (ret < 0) { fprintf(stderr, "Failed to copy context from input to output stream codec context\n"); return -1; } out_stream->codec->codec_tag = 0; if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER) out_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER; } av_dump_format(ofmt_ctx, 0, "test.avi", 1); if (!(ofmt->flags & AVFMT_NOFILE)) { ret = avio_open(&ofmt_ctx->pb, "test.avi", AVIO_FLAG_WRITE); if (ret < 0) { fprintf(stderr, "Could not open output file '%s'", "test.avi"); return -1; } } ret = avformat_write_header(ofmt_ctx, NULL); if (ret < 0) { fprintf(stderr, "Error occurred when opening output file\n"); return -1; } QtConcurrent::run(this, &CameraTest::grabFrames); return 0; } void CameraTest::grabFrames() { AVPacket pkt; int ret; while (av_read_frame(ifmt_ctx, &pkt) >= 0) { AVStream *in_stream, *out_stream; in_stream = ifmt_ctx->streams[pkt.stream_index]; out_stream = ofmt_ctx->streams[pkt.stream_index]; /* copy packet */ pkt.pts = av_rescale_q_rnd(pkt.pts, in_stream->time_base, out_stream->time_base, (AVRounding) (AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX)); pkt.dts = av_rescale_q_rnd(pkt.dts, in_stream->time_base, out_stream->time_base, (AVRounding) (AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX)); pkt.duration = av_rescale_q(pkt.duration, in_stream->time_base, out_stream->time_base); pkt.pos = -1; int ret = av_interleaved_write_frame(ofmt_ctx, &pkt); if (ret < 0) { qDebug() << "Error muxing packet"; //break; } av_free_packet(&pkt); if(done) break; } av_write_trailer(ofmt_ctx); avformat_close_input(&ifmt_ctx); /* close output */ if (ofmt_ctx && !(ofmt->flags & AVFMT_NOFILE)) avio_close(ofmt_ctx->pb); avformat_free_context(ofmt_ctx); if (ret < 0 && ret != AVERROR_EOF) { //return -1; //fprintf(stderr, "Error occurred: %s\n", av_err2str(ret)); } } 

av_interleaved_write_frame returns an error with video packets. The final file shows only the first frame, but the sound looks normal.

On the console, this is what is printed:

 Input #0, dshow, from 'video=Integrated Camera:audio=Microfone interno (Conexant 206': Duration: N/A, start: 146544.738000, bitrate: 1411 kb/s Stream #0:0: Video: rawvideo, bgr24, 640x480, 30 tbr, 10000k tbn, 30 tbc Stream #0:1: Audio: pcm_s16le, 44100 Hz, 2 channels, s16, 1411 kb/s Output #0, avi, to 'test.avi': Stream #0:0: Video: rawvideo, bgr24, 640x480, q=2-31, 30 tbc Stream #0:1: Audio: pcm_s16le, 44100 Hz, 2 channels, s16, 1411 kb/s [avi @ 0089f660] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead. [avi @ 0089f660] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead. [avi @ 0089f660] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 4396365 >= 4396365 [avi @ 0089f660] Too large number of skipped frames 4396359 > 60000 [avi @ 0089f660] Too large number of skipped frames 4396360 > 60000 [avi @ 0089f660] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 4396390 >= 4396390 [avi @ 0089f660] Too large number of skipped frames 4396361 > 60000 [avi @ 0089f660] Too large number of skipped frames 4396362 > 60000 [avi @ 0089f660] Too large number of skipped frames 4396364 > 60000 [avi @ 0089f660] Too large number of skipped frames 4396365 > 60000 [avi @ 0089f660] Too large number of skipped frames 4396366 > 60000 [avi @ 0089f660] Too large number of skipped frames 4396367 > 60000 

It seems to me that this is a simple problem to solve, but I really basically do not know about the ffmpeg API, if someone can lead me in the right direction, it will be great!

Thanks!

Answer 1

Your problem seems a bit specific to DirectShow. Unfortunately, I do not have access to the system with DirectShow, but because of the symptom, it seems that capture is not your problem. What is wrong is the multiplexing part. Maybe the video package format is not directly supported in AVI, or there may be timestamps on the packages.

I will recommend a few things you should try, one at a time:

• Try using av_write_frame instead of av_interleaved_write_frame .
• Use a better container like MP4 or MKV.
• Do not try to connect the input package to the avi file. In grabFrames grab your raw video packets and upload them to a file. This should give you a file that can be played using ffplay. (You will probably need to specify the resolution, format, and image format in the ffplay command.)
• The above result in a playable video file? If so, then I recommend that you decode individual video packets, convert the color space, and encode them using a common codec. (I recommend yuv420p on h264.) The FFmpeg code base has two examples that should be useful - demuxing_decoding.c and decoding_encoding.c . This should give you the correct video file. (Playable for most players.)

I don't know anything about DirectShow, and I don't know your use case. Therefore, my recommendations focus on the FFmpeg API. Some of them may be excessive / may not do what you want.