Tensorflow-Lite preview model not working in Android demo

Question

Tensorflow-Lite Android demo works with the original model that it provides: mobilenet_quant_v1_224.tflite. See: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite

They also provide other pre-prepared Lite models: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/g3doc/models.md

However, I downloaded some of the smaller models from the above link, for example mobilenet_v1_0.25_224.tflite, and replaced the original model with this model in the demo application by simply changing MODEL_PATH = "mobilenet_v1_0.25_224.tflite"; in ImageClassifier.java , Application crash:

12-11 12: 52: 34,222 17713-17729 /? E / AndroidRuntime: FATAL EXCEPTION: CameraBackground Process: android.example.com.tflitecamerademo, PID: 17713 java.lang.IllegalArgumentException: Failed to get input sizes. The 0th input should have 602112 bytes, but 150528 bytes were found. at org.tensorflow.lite.NativeInterpreterWrapper.getInputDims (native Method) on org.tensorflow.lite.NativeInterpreterWrapper.run (NativeInterpreterWrapper.java:82) on org.tensorflow.lite.Interpreter.runForMultiplejputsput1avaOsOrOrOrO1OrOOrOrOOrOrOOrOsOrOUrOsOrOutOrOutOrgetOutOutOrput1 .tensorflow.lite.Interpreter.run (Interpreter.java:93) at com.example.android.tflitecamerademo.ImageClassifier.classifyFrame (ImageClassifier.java:108) at com.example.android.tflitecamerademo.Camera2BasicFragment.classifymentassassassassmentassassment : 663) on com.example.android.tflitecamerademo.Camera2BasicFragment.access $ 900 (Camera2BasicFragment.java:69) at com.example.android.tflitecamerademo.Camera2BasicFragment $ 5.run (Camera2BasicFragment.java UP58) on android.os.Handler.handleCallback (Handler.java:751) on android.os.Handler.dispatchMessage (Handler.java:95) on android.os.Looper.loop (Looper.java:154) on android.os.HandlerThread.run (HandlerThread.java:61)

The reason is that the input size required by the model is four times the size of the image. So I changed DIM_BATCH_SIZE = 1 to DIM_BATCH_SIZE = 4 . Now the error:

FATAL EXCEPTION: CameraBackground Process: android.example.com.tflitecamerademo, PID: 18241 java.lang.IllegalArgumentException: cannot convert a TensorFlowLite tensor of type FLOAT32 to a Java object of type [[B (which is compatible with TensorFlowLite UINT8flow. lite.Tensor.copyTo (Tensor.java:36) at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs (Interpreter.java:122) at org.tensorflow.lite.Interpreter.run (Interpreter.java:93) on com.example .android.tflitecamerademo.ImageClassifier.classifyFrame (ImageClassifier.java:108) at com.example.android.tflitecamerademo.Camera2BasicFragment.classifyFrame (Camera2BasicFragment.java:663) at com.example.android.tflitecamerademo.Camera2BasicFragment.access $ 900ametame.ametame.ametame.ameroid.ame .run (Camera2BasicFragment.java∗58) on android.os.Handler.handleCallback (Handler.java:751) on android.os.Handler.dispatchMessage (Handler.java:95) on android.os.Looper.loop (Looper. java: 154) on android.os.HandlerThread.run (HandlerThread.java:61)

My question is how to get a reduced tplite mobile phone model for working with TF-lite Android Demo.

(I really tried other things, for example, I converted a frozen TF graph into a TF-lite model using the provided tool, even using the same sample code as in https://github.com/tensorflow/tensorflow/blob/master/tensorflow /contrib/lite/toco/g3doc/cmdline_examples.md , but the converted tflite model still cannot work in Android Demo.)

Answer 1

ImageClassifier.java, included in the Tensorflow-Lite Android demo, expects a quantized model. At the moment, only one of the Mobilenets models is presented in quantized form: Mobilenet 1.0 224 Quant.

To use other floating point models, swap them in ImageClassifier.java from the Tensorflow for Poets TF-Lite demo. This is written for float models. https://github.com/googlecodelabs/tensorflow-for-poets-2/blob/master/android/tflite/app/src/main/java/com/example/android/tflitecamerademo/ImageClassifier.java

Make a diff and you will see that there are several important differences in the implementation.

Another consideration is converting floating point models to quantization using TOCO: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/toco/g3doc/cmdline_examples.md

Answer 2

I also got the same errors as Sedling. I created a new image classifier wrapper for the Mobilenet Float model. Now it works great. You can immediately add this class to the image classifier demonstration and use it to create a classifier in Camera2BasicFragment

 classifier = new ImageClassifierFloatMobileNet(getActivity()); 

below is the shell of the classifier of the image classifier for the Mobilenet Float

  /** * This classifier works with the Float MobileNet model. */ public class ImageClassifierFloatMobileNet extends ImageClassifier { /** * An array to hold inference results, to be feed into Tensorflow Lite as outputs. * This isn't part of the super class, because we need a primitive array here. */ private float[][] labelProbArray = null; private static final int IMAGE_MEAN = 128; private static final float IMAGE_STD = 128.0f; /** * Initializes an {@code ImageClassifier}. * * @param activity */ public ImageClassifierFloatMobileNet(Activity activity) throws IOException { super(activity); labelProbArray = new float[1][getNumLabels()]; } @Override protected String getModelPath() { // you can download this file from // https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_224_android_quant_2017_11_08.zip // return "mobilenet_quant_v1_224.tflite"; return "retrained.tflite"; } @Override protected String getLabelPath() { // return "labels_mobilenet_quant_v1_224.txt"; return "retrained_labels.txt"; } @Override public int getImageSizeX() { return 224; } @Override public int getImageSizeY() { return 224; } @Override protected int getNumBytesPerChannel() { // the Float model uses a 4 bytes return 4; } @Override protected void addPixelValue(int val) { imgData.putFloat((((val >> 16) & 0xFF)-IMAGE_MEAN)/IMAGE_STD); imgData.putFloat((((val >> 8) & 0xFF)-IMAGE_MEAN)/IMAGE_STD); imgData.putFloat((((val) & 0xFF)-IMAGE_MEAN)/IMAGE_STD); } @Override protected float getProbability(int labelIndex) { return labelProbArray[0][labelIndex]; } @Override protected void setProbability(int labelIndex, Number value) { labelProbArray[0][labelIndex] = value.byteValue(); } @Override protected float getNormalizedProbability(int labelIndex) { return labelProbArray[0][labelIndex]; } @Override protected void runInference() { tflite.run(imgData, labelProbArray); } }