FFMPEG Seeking Brings Sound Artifacts

Question

I am implementing an audio decoder using ffmpeg. Although reading audio and even searching is already working, I cannot find a way to clear the buffers after the search, so I have no artifacts when the application starts reading audio immediately after the search.

avcodec_flush_buffers does not seem to affect internal buffers. This problem occurs with all decoders (mp3, aac, wma, ...), but PCM / WAV (which does not use internal buffers to store data for decoding, because the sound is not compressed).

The code snippet is simple:

 av_seek_frame(audioFilePack->avContext, audioFilePack->stream, posInTimeFrame, AVSEEK_FLAG_ANY); avcodec_flush_buffers(audioFilePack->avContext->streams[audioFilePack->stream]->codec); 

Clarification:

 audioFilePack->avContext = FormatContext audioFilePack->stream = Stream Position (also used to read audio packets) audioFilePack->avContext->streams[audioFilePack->stream]->codec = CodecContext for the codec used 

Any ideas on what I should do so that I can search and not get any residual sound? Thanks!

Answer 1

This is a bug in ffmpeg. Internal buffers are not flushed, so when you go to receive a packet / frame after flushing, you get preliminary search data. It seems to be fixed from 3-16-12, so you can enable this fix yourself or update ffmpeg.

http://permalink.gmane.org/gmane.comp.video.libav.devel/23455

As an update, the error above is indeed a problem, but there is a second error with AAC.

As of five months ago, another user had discovered this error, and it was reported that it was fixed. https://ffmpeg.org/trac/ffmpeg/ticket/420

A fix is ​​a flash function added to aacdec.c that flushes its internal buffers. The problem is that aacdec.c has two decoders, and only one has been assigned a pointer to the flush function. If you use a different (more general) decoder, it still will not be cleaned properly.

If you are able to build ffmpeg yourself, fix this by adding .flush = flush, at the bottom of the AVCodec definition of ff_aac_decoder (which is at the bottom of the file.)

I will let ffmpeg guys hope so that it can be included in the main branch.

Answer 2

I have never written a searchable audio player, but I suspect this is happening. Each audio packet is decoded into a fragment of the original sound wave. Usually these fragments successively rest against each other, and the result is a continuous wave, which you hear as sound without artifacts. When you search, you force two fragments from disparate parts of the file to rest against each other. This usually leads to a break in the resulting sound wave, which the ear perceives as a click or pop, or as you call it (I guess) an artifact.

Here is a more specific example. Suppose you played the first 25 audio packages before searching. Let's say packet 25 decodes into a wave whose last sample is 12345. While packet 25 is being transmitted to the speaker, you are trying to execute packet 66. Let them say that the first sample of packet 66 is 23456. Thus, the digital audio stream jumps from 12345 to -23456 through the search. This is a huge gap and will be heard as a pop.

I think one solution is to capture one additional packet, before you start searching (packet 26 in my example), decode it in a standalone buffer, apply fade out and then put it in the playback queue. After you try to find the right place, grab the first packet (66 in my eaxmple), decode it in another offline buffer, apply the attenuation, and then put it in the playback queue. This should provide smooth sound waves and search without artifacts.

If you are smart, you can make the fade in and fade out as short or as long as possible. I think that just a few milliseconds should be enough to prevent artifacts. You can even apply cross wilting with old and new packages. It can also be quite simple to mark the last value of the sample in the last packet before the search and gradually transfer it to zero for several samples, rather than immediately pull it to zero. This may be simpler than decoding an additional packet.

This is my guess on how to solve this problem. This is clearly a problem, so I recommend that you also watch the open source audio players and see how they implement the search. Good examples are programs like Audacity, Totem, Banshee, RhythmBox, Amarok, or VLC, or frameworks like GStreamer. If you find that they use well-known methods, please report a topic here. I think people will want to know what they are. Good luck