East tflite on Pixel emulator

[akshyasingh-mset]

I am trying to run EAST tflite model provided by @sayakpaul on tf-hub https://tfhub.dev/sayakpaul/lite-model/east-text-detector/fp16/1 to detect text on an image on Pixel emulator. I am able to use custrom model https://developers.google.com/codelabs/tflite-object-detection-android#6 on sample images to detect salads. I am a complete novice on running tflite on android devices. So i just replaced the above mentioned model with EAST tflite float 16 model and replaced sample image with the image i want east to detect text in. But I am receiving following error :

E/TaskJniUtils: Error getting native address of native library: task_vision_jni java.lang.RuntimeException: Error occurred when initializing ObjectDetector: Input tensor has type kTfLiteFloat32: it requires specifying NormalizationOptions metadata to preprocess input images. at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method) at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149) at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66) at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147) at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130) at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49) at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:184) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-1 Process: org.tensorflow.codelabs.objectdetection, PID: 6881 java.lang.IllegalStateException: Error getting native address of native library: task_vision_jni at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:95) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66) at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147) at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130) at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49) at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:184) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) Caused by: java.lang.RuntimeException: Error occurred when initializing ObjectDetector: Input tensor has type kTfLiteFloat32: it requires specifying NormalizationOptions metadata to preprocess input images. at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method) at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149) at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)  at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147)  at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130)  at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49)  at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:184)  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)  D/EGL_emulation: eglMakeCurrent: 0xcd693de0: ver 3 0 (tinfo 0xe8846f20) W/BaseTaskApi: Closing an already closed native lib

Please help me, I am learning to deploy tflite on mobile devices. Sorry for the english

[sayakpaul]

@farmaker47 could you help here?

[akshyasingh-mset]

@sayakpaul @farmaker47 I used the link : https://www.tensorflow.org/lite/convert/metadata_writer_tutorial to convert the above mentioned float16 tflite EAST model to include metadata. But now I am receiving an altogether different error. It seems I am doing something wrong here. Please help

E/TaskJniUtils: Error getting native address of native library: task_vision_jni java.lang.IllegalArgumentException: Error occurred when initializing ObjectDetector: Mobile SSD models are expected to have exactly 4 outputs, found 2 at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method) at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149) at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66) at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147) at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130) at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49) at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:183) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-1 Process: org.tensorflow.codelabs.objectdetection, PID: 8734 java.lang.IllegalStateException: Error getting native address of native library: task_vision_jni at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:95) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66) at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147) at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130) at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49) at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:183) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) Caused by: java.lang.IllegalArgumentException: Error occurred when initializing ObjectDetector: Mobile SSD models are expected to have exactly 4 outputs, found 2 at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method) at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156) at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149) at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91) at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)  at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147)  at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130)  at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49)  at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:183)  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) 

[farmaker47]

@farmaker47 could you help here?

Yes sure.. Let me take a look in the next days.

[farmaker47]

@singh-aymcommerce Unfortunately object detector applications and EAST tflite file do not match. You have to use a custom project with an Interpreter. Check the guide at Tensorflow Lite https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_java You can firstly use the Python API to run inference..so you can understand how you are going to acomplish that in Java inside android project. https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_python

You have to see the original Python notebook to understand the outputs of the model. there are 2 of them. One of 1,80,80,1 and the other of 1,80,80,5 You have to understand what are those...and how they are converted to desired object

[akshyasingh-mset]

@farmaker47 thank you for your response. I will try your above proposed solution to write custom interpreter in java. Give me couple of days and i will get back if I am able to do it. Thanks a lot @farmaker47 @sayakpaul :D

[akshyasingh-mset]

@sayakpaul @farmaker47 I am successfully able to import tflite model for EAST with your help.

i am coming across one weird thing that while adding metadata i added a labels.txt with only one line "text" as only one class is detected as label

but there are all null values in the geometry array when i try to inference an image with text PFA. But above tflite file is working when i try to detect text boxes using @sayakpaul Python script. The EAST tflite model is expecting input in float32 format. Am i doing something wrong, please suggest

dtype = {DataType@10768} "FLOAT32" nativeHandle = 3391991168 quantizationParamsCopy = {Tensor$QuantizationParams@10769} shapeCopy = {int[4]@10770} [1, 320, 320, 3] shapeSignatureCopy = {int[4]@10771} [1, 320, 320, 3] shadow$klass = {Class@10688} "class org.tensorflow.lite.Tensor" shadow$monitor = 0

[akshyasingh-mset]

@sayakpaul @farmaker47 I have solved the above issue :D

I have come across another roadblock on how to implement nonMaxSuppresion on android. Can you help me please?

[10770]

Boa noite, eu acho que você está enviando um e-mail para a pessoa errada. Sou José Felipe, sou do Brasil.

Em qua., 29 de dez. de 2021 às 05:17, singh-aymcommerce < @.***> escreveu:

@sayakpaul https://github.com/sayakpaul @farmaker47 https://github.com/farmaker47 I have solved the above issue :D

I have come across another roadblock on how to implement nonMaxSuppresion on android. Can you help me please?

— Reply to this email directly, view it on GitHub https://github.com/ml-gde/e2e-tflite-tutorials/issues/56#issuecomment-1002451407, or unsubscribe https://github.com/notifications/unsubscribe-auth/AV2VUNDEYVTRS7BE3YF64B3UTK727ANCNFSM5JQ6BQFA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

--

*Cordialmente, * *José Felipe. *

Universidade Federal de Sergipe (UFS), Campus São Cristóvão. Departamento de Química (DQI) Laboratório de Corrosão e Nanotecnologia (LCNT/NUPEG). Discente do curso de Química licenciatura. Segundo Secretário Geral do CALIQ. Contato: (79) 999889473, @.*** Currículo Lattes: http://lattes.cnpq.br/5234169420338145