OpenCV Android - color issue using CameraBridgeViewBase

Question

I am having a strange problem using Android emulators and OpenCV CameraBridgeViewBase .

Using onCameraFrame , I get an image that looks like it was not decoded correctly.

 public Mat onCameraFrame(CvCameraViewFrame inputFrame) { return inputFrame.rgba(); } 

Using 'inputFrame.gray ()' I get the expected - black and white image without artifacts or any other problems.

What I get:

Another picture (large)

What I have tried so far:

• Different API levels (15 to 21).
• Various emulators: Genymotion and Google Android emulator.
• Various platform architectures - both ARM and Intel x86.
• Running the emulator on another laptop with Linux: it works , as expected, the problem disappeared!
• Launch applications using OpenCV downloaded from the Play Store. They work! However:
  • Launch an application that works as expected, then close it.
  • Launch the application (or one of the OpenCV tutorials), then close it.
  • Starting the application from 5.1 again, I see that it affected the same error!
• Different versions of OpenCV (versions 2.4.9 and 2.4.10).
• Different versions of the OpenCV manager (one of the Play Store and 2.4.9 and 2.4.10 from the OpenCV package).
• Finally, as I noted in 5.2, the precompiled training .apk files from the OpenCV package are also affected.

Everything works as expected on my real Android devices.

Looking at the sources of the CameraBridgeViewBase and Java / Native classes, I came to the CameraBridgeViewBase that the problem occurs when decoding the image. There is probably a problem with the output format of the camera on the platform (YUV, NV21). However, it is strange that .gray () produces the correct image (without artifacts).

I use Mac OS X 10.10 Yosemite and MacBook Air with a Facetime HD camera, if that matters.

Any ideas on how to overcome this problem and help find the root of the problem are welcome!

Answer 1

So, after drilling the problem, I found the root of the problem.

Let's look at the OpenCV class JavaCameraView and its base class CameraBridgeViewBase . The problem was that the camera frames received as the byte[] array in onPreviewFrame were incorrectly decoded.

The exact place where the decoding process takes place is the implementation of the Mat rgba() method in the internal JavaCameraFrame class of JavaCameraView :

  public Mat rgba() { Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4); return mRgba; } 

As we can see, the Imgproc.cvtColor (...) method is used to convert a frame from YUV to RGBA. The NV21 YUV -> RGBA conversion takes place there. During the initialization process, we set the format to NV21, so this should be correct. In addition, each Android device must support NV21 . In addition, we can check if the device has adopted the format using the debugger:

 protected boolean initializeCamera(int width, int height) { ... params.setPreviewFormat(ImageFormat.NV21); ... mCamera.setParameters(params); ... params = mCamera.getParameters(); Log.d(TAG, String.format("Actual preview format is 0x%X", params.getPreviewFormat())); } 

Both phones (HTC Sensation) and the emulator reported using NV21.

However, if we change COLOR_YUV2RGBA_NV21 to COLOR_YUV2RGB_I420 (YV12 and I420 are the same, only with inverted Y and V;), we will see that the emulator finally gets the correct color space. Changing NV21 to YV12 in params.setPreviewFormat(ImageFormat.NV21); , we get similar results. Looks like an error in Imgproc.cvtColor or on Android.

Here is the solution. Change the public Mat rgba() as follows:

  public Mat rgba() { if (previewFormat == ImageFormat.NV21) { Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4); } else if (previewFormat == ImageFormat.YV12) { Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGB_I420, 4); // COLOR_YUV2RGBA_YV12 produces inverted colors } return mRgba; } 

previewFormat is a new int variable that is declared as follows:

 private int previewFormat = ImageFormat.NV21; 

Add the following initialization changes:

 protected boolean initializeCamera(int width, int height) { ... params.setPreviewFormat(ImageFormat.NV21); // "generic" = android emulator if (Build.BRAND.equalsIgnoreCase("generic")) { params.setPreviewFormat(ImageFormat.YV12); } ... mCamera.setParameters(params); params = mCamera.getParameters(); previewFormat = params.getPreviewFormat(); ... } 

Important:
Please note: this is just a temporary solution to make OpenCV usable with the emulator in my case. Further studies should be performed. It's easy enough to check if your device is using the correct image format in onPreviewFrame. I will come back to this when I have time.

Answer 2

For those of you who use genymotion, BRAND will not be called generic. To solve this problem, just change the code to

 ... if (Build.BRAND.equalsIgnoreCase("generic") | Build.BRAND.equalsIgnoreCase("Android")) { ... 

and it should work or just print the BRAND that you use and replace it with "Android"

PS The second condition will include the case when you use genymotion, which is also an emulator.

Greetings

Answer 3

This issue has now been fixed in the code in git since the Pull request https://github.com/opencv/opencv/pull/8168 has been merged. It should be available in the next released version.