What is the AVFoundation AVAssetWriterInput expects the MediaDataInRealTime property?

Question

I am working on an application that reads multimedia data from a disk, converts it to the appropriate pixel format, and then transfers it to AVAssetWriter for compression and writing to disk. I am doing my reading interleaved and not using AVAssetReader. My reader guarantees that a single-frame video and audio data with one frame will be presented in a serial way. The problem I have is, if I do not set the expectsMediaDataInRealTime property to Yes , the author of the video asset will always return NO for isReadyForMoreMediaData exactly after 30 frames. If I stop writing up to 30 frames, it works fine and the output file is valid. However, if I set expectMediaDataInRealTime to YES, it works fine all the time, which can be several thousand frames. After starting the transcoding operation, waiting for the MediaDataInRealTime parameter set to YES, I observed the application's memory usage during compression of a very long video, and there was no unreasonable memory usage and memory leaks. And the resulting MOV file looked quite normal, for example. audio data alternates with video data, as you would expect.

So, why should I always expect MediaDataInRealTime to be NO if there is no obvious flaw to set it to YES? Is this only used when using the Apple API to read data (using AVAssetReader)? The documentation says that this property control writes "multimedia data in an ideal interleaving pattern for efficient storage and playback", but when expectsMediaDataInRealTime is set to YES, isReadyForMoreMediaData never returns NO and a file appears that will be written perfectly. So, if AVAssetWriter can do this when this property is set to YES, why can't it do this when set to NO? The source is exactly the same.

What exactly does this property do except "make sure that the readyForMoreMediaData value is properly calculated" (which means absolutely nothing to me)?

Answer 1

As I understand it, the installation expects that MediaDataInRealTime - YES means that the encoder is waiting for real-time data transmission, for example, cameras, etc. In this case, you will constantly feed the encoder with data, isReadyForMoreMediaData will tell you if it can feed data to the encoder. If isReadyForMoreMediaData is NO, you will have to drop the current pattern and wait until the next pattern appears, and check isReadyForMoreMediaData again for whether it is YES.

On the other hand, if you expect that MediaDataInRealTime is NO, it means that the encoder does not use the source in real time, but an autonomous data stream, for example, AVAssetReader. In this case, since you can control the flow rate yourself, so there is no NO on isReadyForMoreMediaData, you can hold the input and encoder and wait until isReadyForMoreMediaData turns into YES (for example, use an infinite loop to sleep and wait for the isReadyForMoreMediaData to change, etc.) .

The purpose of this, I believe, is that for the internal mechanism, you can try to alternate the timestamp of audio and video as even as possible, so players, decoders, do not need to pre-sample a large amount of data for playback. This is a compromise between the integrity of the data of the final data source and the experience of reproduction. Although, I suppose, you can use expectsMediaDataInRealTime for YES all the time, but it is better to discard early samples in the case of isReadyForMoreMediaData == NO.