All geek questions in one place

Mp4 / h.264 decoding using MediaCodec without MediaExtractor, expected access block format

I am trying to use the MediaCodec API to decode without using the MediaExtractor API. Instead, I use mp4parser to get samples from mp4 files. At the moment, I only use h.264 / avc encoded video.

The official MediaCodec API documentation states:

buffers do not start and end at arbitrary byte boundaries, this is not a stream of bytes, this is a stream of access blocks.

Meaning, I need to pass access blocks to the decoder. However, I will skip some details of this information:

For h.264, there can be several NAL blocks in the mp4 sample, each of which is preceded by 4 bytes (default), which determine the length of the NAL block.

Now my questions are:

  • There may be mp4 samples where the codec's NAL codes (sps, pps) are mixed with NAL units containing encoded (parts) of the frames. In this case, should I pass the BUFFER_FLAG_CODEC_CONFIG flag when calling queueInputBuffers() ?

  • There may also be other (optional) NAL units in mp4 samples, such as NAL units of the SEI access unit or access unit. What is the matter with them? No problems?

I tried different possibilities, but all the feedback I get from Android is that the calls to dequeueOutputBuffer() time out (or don't return if I pass -1 as a timeout parameter). As a result, I have no way to fix this problem.

Any advice on what to do or where to look is, of course, very welcome.

+6
source share
2 answers

The NAL length prefix that indicates the length of the NAL block must be converted to the application-B initial codes (bytes 0x00, 0x00, 0x00, 0x01) before proceeding to MediaCodec for decoding. (Some decoders can immediately accept the MP4 format, but this is not too common.)

SPS / PPS, which is stored in the avcC atom in a file, also needs to be converted in order to use the application-B initial codes. Please note that avcC atom contains several other fields that you do not need to pass to the decoder. You can transfer SPS and PPS in one buffer (with initial codes in front of each of them) with the BUFFER_FLAG_CODEC_CONFIG flag before sending any real frames or transfer them (using the initial codes of application-B) to MediaFormat, which you use to configure the decoder (or in one ByteBuffer with the key "csd-0", or two separate keys "csd-0" and "csd-1").

If your file has more SPS / PPS inside each frame, you should simply transfer them as part of the frame, and most decoders should be able to handle it (especially if it is the same SPS / PPS as before and after the configuration change).

Thus: transfer all NAL units belonging to the same sample in the same buffer, but all the NAL header headers are overwritten with initial codes. And to work with MP4 files that do not have SPS / PPS inside the stream itself, analyze avcC atom (I don’t know in what format mp4parser returns this) and transfer SPS and PPS with initial codes to the decoder (or via MediaFormat as "csd- 0 ", or as the first buffer with the setting BUFFER_FLAG_CODEC_CONFIG).

+6
source

Getting -1 is the usual simple decoding, you should see something on the screen. Until it throws an IllegalState exception, just keep decoding

0
source

Source: https://habr.com/ru/post/973419/

More articles:


All Articles