The built-in window queueBuffer function does not output from the Stagefright decoder

Question

I pass a SurfaceView surface from Java to JNI, where I get my own window from that surface. Stagefright decodes h264 frames from the mp4 file. During the decoding process, I call ANativeWindow::queueBuffer() to send the decoded frames for rendering. There are no errors when decoding or when calling queueBuffer() , all I get is a black screen.

It really seems to me that I'm not setting up my own window properly, so when queueBuffer() is queueBuffer() , it is displayed on the screen. However, I can display pixels in my own window directly through memcpy. Unfortunately, after creating the OMXClient instance, a segfault attempt occurs while trying to draw pixels manually, so it seems to me that I should use queueBuffer() .

My surface setting in onCreate ():

 protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); SurfaceView surfaceView = new SurfaceView(this); surfaceView.getHolder().addCallback(this); setContentView(surfaceView); } 

Once the surface is created, I call my own init () function with the surface:

 @Override public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { NativeLib.init(holder.getSurface(), width, height); } 

JNI creates its own window and begins the decoding stream:

 nativeWindow = ANativeWindow_fromSurface(env, surface); int ret = pthread_create(&decode_thread, NULL, &decode_frames, NULL); 

My frame decoding routine a la vec.io Stagefright decoding example

 void* decode_frames(void*){ mNativeWindow = nativeWindow; sp<MediaSource> mVideoSource = new AVFormatSource(); OMXClient mClient; mClient.connect(); sp<MediaSource> mVideoDecoder = OMXCodec::Create(mClient.interface(), mVideoSource->getFormat(), false, mVideoSource, NULL, 0, mNativeWindow); mVideoDecoder->start(); while(err != ERROR_END_OF_STREAM ) { MediaBuffer *mVideoBuffer; MediaSource::ReadOptions options; err = mVideoDecoder->read(&mVideoBuffer, &options); if (err == OK) { if (mVideoBuffer->range_length() > 0) { sp<MetaData> metaData = mVideoBuffer->meta_data(); int64_t timeUs = 0; metaData->findInt64(kKeyTime, &timeUs); status_t err1 = native_window_set_buffers_timestamp(mNativeWindow.get(), timeUs * 1000); //This line results in a black frame status_t err2 = mNativeWindow->queueBuffer(mNativeWindow.get(), mVideoBuffer->graphicBuffer().get(), -1); if (err2 == 0) { metaData->setInt32(kKeyRendered, 1); } } mVideoBuffer->release(); } } mVideoSource.clear(); mVideoDecoder->stop(); mVideoDecoder.clear(); mClient.disconnect(); } 

EDIT : Accepting Ganesha's advice, I contacted Awesome Renderer to change the color space. During this, it became apparent that the color format was not set in Stagefright.

 08-06 00:56:32.842: A/SoftwareRenderer(7326): frameworks/av/media/libstagefright/colorconversion/SoftwareRenderer.cpp:42 CHECK(meta->findInt32(kKeyColorFormat, &tmp)) failed. 08-06 00:56:32.842: A/libc(7326): Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1), thread 7340 (hieu.alloclient) 

Trying to set the color space explicitly (kKeyColorFormat to the yuv420P color space) leads to the decompression problem. Which probably makes sense because the color format you specify is arbitrary.

 08-06 00:44:30.878: V/OMXCodec(6937): matchComponentName (null) 08-06 00:44:30.888: V/OMXCodec(6937): matching 'OMX.qcom.video.decoder.avc' quirks 0x000000a8 08-06 00:44:30.888: V/OMXCodec(6937): matchComponentName (null) 08-06 00:44:30.888: V/OMXCodec(6937): matching 'OMX.google.h264.decoder' quirks 0x00000000 08-06 00:44:30.888: V/OMXCodec(6937): Attempting to allocate OMX node 'OMX.qcom.video.decoder.avc' 08-06 00:44:30.918: V/OMXCodec(6937): Successfully allocated OMX node 'OMX.qcom.video.decoder.avc' 08-06 00:44:30.918: V/OMXCodec(6937): configureCodec protected=0 08-06 00:44:30.918: I/OMXCodec(6937): [OMX.qcom.video.decoder.avc] AVC profile = 66 (Baseline), level = 13 08-06 00:44:30.918: V/OMXCodec(6937): [OMX.qcom.video.decoder.avc] setVideoOutputFormat width=320, height=240 08-06 00:44:30.918: V/OMXCodec(6937): [OMX.qcom.video.decoder.avc] portIndex: 0, index: 0, eCompressionFormat=7 eColorFormat=0 08-06 00:44:30.918: V/OMXCodec(6937): [OMX.qcom.video.decoder.avc] found a match. 08-06 00:44:30.938: I/QCOMXCodec(6937): Decoder should be in arbitrary mode 08-06 00:44:30.958: I/OMXCodec(6937): [OMX.qcom.video.decoder.avc] video dimensions are 320 x 240 08-06 00:44:30.958: I/OMXCodec(6937): [OMX.qcom.video.decoder.avc] Crop rect is 320 x 240 @ (0, 0) 08-06 00:44:30.958: D/infoJNI(6937): before started 08-06 00:44:30.968: V/OMXCodec(6937): [OMX.qcom.video.decoder.avc] allocating 2 buffers of size 2097088 on input port 08-06 00:44:30.968: V/OMXCodec(6937): [OMX.qcom.video.decoder.avc] allocated buffer 0x417037d8 on input port 08-06 00:44:30.968: V/OMXCodec(6937): [OMX.qcom.video.decoder.avc] allocated buffer 0x41703828 on input port 08-06 00:44:30.978: V/OMXCodec(6937): native_window_set_usage usage=0x40000000 08-06 00:44:30.978: V/OMXCodec(6937): [OMX.qcom.video.decoder.avc] allocating 22 buffers from a native window of size 147456 on output port 08-06 00:44:30.978: E/OMXCodec(6937): dequeueBuffer failed: Invalid argument (22) 

Answer 1

I decided to solve this problem using the low level Java APIs. I installed my own read_frame function, which parses video frames using FFmpeg . I call this function in a separate Java decoder thread that returns a new data frame that should be decoded by MediaCodec . It was very simple to do so: just walk through the MediaCodec surface.

Alternatively, I could use MediaExtractor , but FFmpeg had some other features that I needed.

Answer 2

Just in case, the problem is not solved! I had the same problem and accidentally discovered a problem.

@Override public void surfaceChanged (holder SurfaceHolder, format int, width int, int height) {NativeLib.init (holder.getSurface (), width, height); }

you need to allocate a frame buffer for a size divisible by 16, which is the size of the macroblock. Otherwise, the graphics buffer is not enough to decode the output. The H264 encoder has an internal small large frame size for the encoding process if the provided video sequence has a width or height that is not aligned with the macro block. Just apply the following: width = 16 * (width + 15) / 16; height = 16 * (height + 15) / 16;

Answer 3

You need to call native_window_set_scaling_mode (mNativeWindow-> get (), NATIVE_WINDOW_SCALING_MODE_SCALE_TO_WINDOW);