All geek questions in one place

Android: SurfaceTexure, camera latency

I try to use MediaCodec and MediaMux, and I am encountering some problems.

Here are the errors from logcat:

12-13 11:59:58.238: E/AndroidRuntime(23218): FATAL EXCEPTION: main 12-13 11:59:58.238: E/AndroidRuntime(23218): java.lang.RuntimeException: Unable to resume activity {com.brendon.cameratompeg/com.brendon.cameratompeg.CameraToMpeg}: java.lang.IllegalStateException: Can't stop due to wrong state. 12-13 11:59:58.238: E/AndroidRuntime(23218): at android.app.ActivityThread.performResumeActivity(ActivityThread.java:2918)

The code is erroneous in "mStManager.awaitNewImage ();", which is in the onResume function. And the logarithm says "camera frame timeout."
mStManager is an instance of the SurfaceTextureManager class. And the "camera frame timeout" comes from the awaitNewImage () function. I have added this class to my post.

Part of my code looks like this (onCreate function and onResume function):

  @Override protected void onCreate(Bundle savedInstanceState) { // arbitrary but popular values int encWidth = 640; int encHeight = 480; int encBitRate = 6000000; // Mbps Log.d(TAG, MIME_TYPE + " output " + encWidth + "x" + encHeight + " @" + encBitRate); super.onCreate(savedInstanceState); setContentView(R.layout.activity_camera_to_mpeg); prepareCamera(encWidth, encHeight); prepareEncoder(encWidth, encHeight, encBitRate); mInputSurface.makeCurrent(); prepareSurfaceTexture(); mCamera.startPreview(); } @Override public void onResume(){ try { long startWhen = System.nanoTime(); long desiredEnd = startWhen + DURATION_SEC * 1000000000L; SurfaceTexture st = mStManager.getSurfaceTexture(); int frameCount = 0; while (System.nanoTime() < desiredEnd) { // Feed any pending encoder output into the muxer. drainEncoder(false); // Switch up the colors every 15 frames. Besides demonstrating the use of // fragment shaders for video editing, this provides a visual indication of // the frame rate: if the camera is capturing at 15fps, the colors will change // once per second. if ((frameCount % 15) == 0) { String fragmentShader = null; if ((frameCount & 0x01) != 0) { fragmentShader = SWAPPED_FRAGMENT_SHADER; } mStManager.changeFragmentShader(fragmentShader); } frameCount++; // Acquire a new frame of input, and render it to the Surface. If we had a // GLSurfaceView we could switch EGL contexts and call drawImage() a second // time to render it on screen. The texture can be shared between contexts by // passing the GLSurfaceView EGLContext as eglCreateContext() share_context // argument. mStManager.awaitNewImage(); mStManager.drawImage(); // Set the presentation time stamp from the SurfaceTexture time stamp. This // will be used by MediaMuxer to set the PTS in the video. if (VERBOSE) { Log.d(TAG, "present: " + ((st.getTimestamp() - startWhen) / 1000000.0) + "ms"); } mInputSurface.setPresentationTime(st.getTimestamp()); // Submit it to the encoder. The eglSwapBuffers call will block if the input // is full, which would be bad if it stayed full until we dequeued an output // buffer (which we can't do, since we're stuck here). So long as we fully drain // the encoder before supplying additional input, the system guarantees that we // can supply another frame without blocking. if (VERBOSE) Log.d(TAG, "sending frame to encoder"); mInputSurface.swapBuffers(); } // send end-of-stream to encoder, and drain remaining output drainEncoder(true); } catch(Exception e) { Log.d(TAG, e.getMessage()); // release everything we grabbed releaseCamera(); releaseEncoder(); releaseSurfaceTexture(); } }

the class in the code that relates to the error

  private static class SurfaceTextureManager implements SurfaceTexture.OnFrameAvailableListener { private SurfaceTexture mSurfaceTexture; private CameraToMpeg.STextureRender mTextureRender; private Object mFrameSyncObject = new Object(); // guards mFrameAvailable private boolean mFrameAvailable; /** * Creates instances of TextureRender and SurfaceTexture. */ public SurfaceTextureManager() { mTextureRender = new CameraToMpeg.STextureRender(); mTextureRender.surfaceCreated(); if (VERBOSE) Log.d(TAG, "textureID=" + mTextureRender.getTextureId()); mSurfaceTexture = new SurfaceTexture(mTextureRender.getTextureId()); // This doesn't work if this object is created on the thread that CTS started for // these test cases. // // The CTS-created thread has a Looper, and the SurfaceTexture constructor will // create a Handler that uses it. The "frame available" message is delivered // there, but since we're not a Looper-based thread we'll never see it. For // this to do anything useful, OutputSurface must be created on a thread without // a Looper, so that SurfaceTexture uses the main application Looper instead. // // Java language note: passing "this" out of a constructor is generally unwise, // but we should be able to get away with it here. mSurfaceTexture.setOnFrameAvailableListener(this); } public void release() { // this causes a bunch of warnings that appear harmless but might confuse someone: // W BufferQueue: [unnamed-3997-2] cancelBuffer: BufferQueue has been abandoned! //mSurfaceTexture.release(); mTextureRender = null; mSurfaceTexture = null; } /** * Returns the SurfaceTexture. */ public SurfaceTexture getSurfaceTexture() { return mSurfaceTexture; } /** * Replaces the fragment shader. */ public void changeFragmentShader(String fragmentShader) { mTextureRender.changeFragmentShader(fragmentShader); } /** * Latches the next buffer into the texture. Must be called from the thread that created * the OutputSurface object. */ public void awaitNewImage() { final int TIMEOUT_MS = 2500; synchronized (mFrameSyncObject) { while (!mFrameAvailable) { try { // Wait for onFrameAvailable() to signal us. Use a timeout to avoid // stalling the test if it doesn't arrive. mFrameSyncObject.wait(TIMEOUT_MS); if (!mFrameAvailable) { // TODO: if "spurious wakeup", continue while loop throw new RuntimeException("Camera frame wait timed out"); } } catch (InterruptedException ie) { // shouldn't happen throw new RuntimeException(ie); } } mFrameAvailable = false; } // Latch the data. mTextureRender.checkGlError("before updateTexImage"); mSurfaceTexture.updateTexImage(); } /** * Draws the data from SurfaceTexture onto the current EGL surface. */ public void drawImage() { mTextureRender.drawFrame(mSurfaceTexture); } @Override public void onFrameAvailable(SurfaceTexture st) { if (VERBOSE) Log.d(TAG, "new frame available"); synchronized (mFrameSyncObject) { if (mFrameAvailable) { throw new RuntimeException("mFrameAvailable already set, frame could be dropped"); } mFrameAvailable = true; mFrameSyncObject.notifyAll(); } } }

Does anyone have any ideas? Thanks!

+3
source share
2 answers

I ran into this problem. The reason is because your code is running in a looper thread. You need to make sure that the code runs on a thread that does not have a looper. If so, SurfaceTexture.OnFrameAvailableListener delivers the "frame available" message to the waiting thread, rather than sending the message to the main thread handler, and you get stuck.

The Bigflake examples provide you with a detailed description:

 /** * Wraps testEditVideo, running it in a new thread. Required because of the way * SurfaceTexture.OnFrameAvailableListener works when the current thread has a Looper * configured. */ private static class VideoEditWrapper implements Runnable { private Throwable mThrowable; private DecodeEditEncodeTest mTest; private VideoEditWrapper(DecodeEditEncodeTest test) { mTest = test; } @Override public void run() { try { mTest.videoEditTest(); } catch (Throwable th) { mThrowable = th; } } /** Entry point. */ public static void runTest(DecodeEditEncodeTest obj) throws Throwable { VideoEditWrapper wrapper = new VideoEditWrapper(obj); Thread th = new Thread(wrapper, "codec test"); th.start(); th.join(); if (wrapper.mThrowable != null) { throw wrapper.mThrowable; } } }
+5
source

As Florian correctly explained, the problem is that your code is executing on a thread that has a looper. You must make sure that the code is executed in a thread that does not have a looper.

I solved this by changing the setup () method in OutputSurface and making sure that setOnFrameListener () is attached to another handler thread.

Here is the code for the same:

 class OutputSurface implements SurfaceTexture.OnFrameAvailableListener { private static final String TAG = "OutputSurface"; private static final boolean VERBOSE = false; private EGLDisplay mEGLDisplay = EGL14.EGL_NO_DISPLAY; private EGLContext mEGLContext = EGL14.EGL_NO_CONTEXT; private EGLSurface mEGLSurface = EGL14.EGL_NO_SURFACE; private SurfaceTexture mSurfaceTexture; private Surface mSurface; private Object mFrameSyncObject = new Object(); private boolean mFrameAvailable; private TextureRender mTextureRender; private HandlerThread mHandlerThread; private Handler mHandler; public OutputSurface(int width, int height) { if (width <= 0 || height <= 0) { throw new IllegalArgumentException(); } eglSetup(width, height); makeCurrent(); setup(); } public OutputSurface() { setup(); } private void setup() { mTextureRender = new TextureRender(); mTextureRender.surfaceCreated(); mHandlerThread = new HandlerThread("callback-thread"); mHandlerThread.start(); mHandler = new Handler(mHandlerThread.getLooper()); // Even if we don't access the SurfaceTexture after the constructor returns, we // still need to keep a reference to it. The Surface doesn't retain a reference // at the Java level, so if we don't either then the object can get GCed, which // causes the native finalizer to run. if (VERBOSE) Log.d(TAG, "textureID=" + mTextureRender.getTextureId()); mSurfaceTexture = new SurfaceTexture(mTextureRender.getTextureId()); // This doesn't work if OutputSurface is created on the thread that CTS started for // these test cases. // // The CTS-created thread has a Looper, and the SurfaceTexture constructor will // create a Handler that uses it. The "frame available" message is delivered // there, but since we're not a Looper-based thread we'll never see it. For // this to do anything useful, OutputSurface must be created on a thread without // a Looper, so that SurfaceTexture uses the main application Looper instead. // // Java language note: passing "this" out of a constructor is generally unwise, // but we should be able to get away with it here. if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) { mSurfaceTexture.setOnFrameAvailableListener(this, mHandler); } else { mSurfaceTexture.setOnFrameAvailableListener(this); } mSurface = new Surface(mSurfaceTexture); } }

The rest of the OutputSurface class may remain the same.

0
source

Source: https://habr.com/ru/post/1205187/

More articles:


All Articles