System error displaying MediaProjection virtual display output in ImageReader

Question

I am working on an application that needs to capture a screen into a bitmap for transmission. I am trying to use the new Android 5.0 API android.media.projection to perform screen capture.

The workflow for this API ends with a call.

mediaProjection.createVirtualDisplay("Test Screen", WIDTH, HEIGHT, DPI, DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR, surface, null, null); 

In my initial attempt at this capture, I got a surface object from SurfaceView. This works correctly; the end result is a tiny duplicate of the display on the screen (the result is the effect of the droost )

I thought the function was almost complete, but I then discovered that SurfaceViews (in terms of code) are not readable; You cannot get a bitmap from them.

Looking for other solutions, I came across this question , which has a very similar purpose for mine, and in this thread it was suggested to use ImageReader instead of SurfaceView for the Surface source that you pass to the createVirtualDisplay API call.

However, when I change my code to use ImageReader instead of SurfaceView, I get logcat errors at runtime (with no exceptions), and the callback function for ImageReader is never called. The createVirtualDisplay call also returns the visible virtual object VirtualDisplay.

Here is logcat:

 9230-9270/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: createGraphicBuffer failed 9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count 9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count 9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count 9230-9246/com.android.techrocket9.nanoid E/BufferQueueProducer﹕ [unnamed-9230-0] dequeueBuffer: can't dequeue multiple buffers without setting the buffer count 

This second line is repeated ~ 100 times before it stops occurring.

After going through the debugger, I see that the first error occurs during a call to createVirtualDisplay, and all the others occur after execution returns to the system code.

only the significant result for this error relates to a problem in Kitkat where the API I'm trying to use does not exist. However, I tried the fix suggested here (putting android:hardwareAccelerated="false" in the manifest). This did not change the behavior of the application.

How can I "set the buffer counter" or otherwise bypass this error and get the screen as a bitmap?

PS My development platform is Nexus 6.

Full code as requested:

 MediaProjection mediaProjection = mgr.getMediaProjection(resultCode, data); ImageReader ir = ImageReader.newInstance(WIDTH, HEIGHT, ImageFormat.JPEG, 5); VirtualDisplay v = mediaProjection.createVirtualDisplay("Test Screen", WIDTH, HEIGHT, getApplicationContext().getResources().getDisplayMetrics().densityDpi, DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR, ir.getSurface(), null, null); 

Edit: Regarding the problem with the artifact, here is the code I use to get the bitmap from the image and display it:

  public void onImageAvailable(ImageReader reader) { Image image = null; ByteArrayOutputStream bos = null; try { image = reader.acquireLatestImage(); if (null == image){ return; } bos = new ByteArrayOutputStream(); final Image.Plane[] planes = image.getPlanes(); final ByteBuffer buffer = (ByteBuffer) planes[0].getBuffer().rewind(); final Bitmap bitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(), Bitmap.Config.ARGB_8888); bitmap.copyPixelsFromBuffer(buffer); //bitmap.compress(Bitmap.CompressFormat.WEBP, 50, bos); runOnUiThread(new Runnable() { public void run() { iv.setImageBitmap(bitmap); } }); 

Answer 1

I think I can answer this question now, I met the same problem, and after I changed ImageFormat.JPEG to PixelFormat.RGBA_8888 , everything will be fine. ImageFormat.JPEG seems to be not supported.

To get the correct bitmap, you need to use the following code:

  int width = img.getWidth(); int height = img.getHeight(); int pixelStride = planes[0].getPixelStride(); int rowStride = planes[0].getRowStride(); int rowPadding = rowStride - pixelStride * width; byte[] newData = new byte[width * height * 4]; int offset = 0; bitmap = Bitmap.createBitmap(metrics,width, height, Bitmap.Config.ARGB_8888); ByteBuffer buffer = planes[0].getBuffer(); for (int i = 0; i < height; ++i) { for (int j = 0; j < width; ++j) { int pixel = 0; pixel |= (buffer.get(offset) & 0xff) << 16; // R pixel |= (buffer.get(offset + 1) & 0xff) << 8; // G pixel |= (buffer.get(offset + 2) & 0xff); // B pixel |= (buffer.get(offset + 3) & 0xff) << 24; // A bitmap.setPixel(j, i, pixel); offset += pixelStride; } offset += rowPadding; } 

So the content of the bitmap is what you want.

PS: I really want to say that the android dock is pretty bad. we need to examine too many details to use sdk api correctly.

Answer 2

The best way to get an image from ImageReader is to simply create a bitmap with the correct size and use the copyPixelsFromBuffer () method. Create an ImageReader as follows:

 mImageReader = ImageReader.newInstance(mWidth, mHeight, ImageFormat.RGB_565, 2); 

Then you can get the image from mImageReader using the following code.

 final Image.Plane[] planes = image.getPlanes(); final ByteBuffer buffer = planes[0].getBuffer(); int offset = 0; int pixelStride = planes[0].getPixelStride(); int rowStride = planes[0].getRowStride(); int rowPadding = rowStride - pixelStride * mWidth; // create bitmap bitmap = Bitmap.createBitmap(mWidth+rowPadding/pixelStride, mHeight, Bitmap.Config.RGB_565); bitmap.copyPixelsFromBuffer(buffer); image.close(); 

I described the screen capture process using the MediaProjection API, along with the mistakes that most people made when retrieving an image from ImageReader on a blog , which you can read if you want.