React Native real time object detection is too slow

[fazzysyed]

import { useState, useRef } from "react"; import { View } from "react-native"; import { Camera, CameraType } from "expo-camera"; import { GLView } from "expo-gl"; import Expo2DContext from "expo-2d-context"; import * as tf from "@tensorflow/tfjs"; import { cameraWithTensors } from "@tensorflow/tfjs-react-native"; import { preprocess } from "../utils/preprocess"; import { renderBoxes } from "../utils/renderBox";

const TensorCamera = cameraWithTensors(Camera);

const CameraView = ({ type, model, inputTensorSize, config, children }) => { const [ctx, setCTX] = useState(null); const typesMapper = { back: CameraType.back, front: CameraType.front };

const cameraStream = (images) => { const detectFrame = async () => { tf.engine().startScope(); const [input, xRatio, yRatio] = preprocess( images.next().value, inputTensorSize[2], inputTensorSize[1] ); await model.executeAsync(input).then((res) => { const [boxes, scores, classes] = res.slice(0, 3); const boxes_data = boxes.dataSync(); const scores_data = scores.dataSync(); const classes_data = classes.dataSync();

    renderBoxes(
      ctx,
      config.threshold,
      boxes_data,
      scores_data,
      classes_data,
      [xRatio, yRatio]
    );
    tf.dispose([res, input]);
  });
  tf.engine().endScope();

  requestAnimationFrame(detectFrame);
};

detectFrame();

};

return ( <>

  {ctx && (
    <TensorCamera
      // Standard Camera props
      className="w-full h-full z-0"
      type={typesMapper[type]}
      // Tensor related props
      //use_custom_shaders_to_resize={true}
      resizeHeight={inputTensorSize[1]}
      resizeWidth={inputTensorSize[2]}
      resizeDepth={inputTensorSize[3]}
      onReady={cameraStream}
      autorender={true}
    />
  )}
  <View className="absolute left-0 top-0 w-full h-full flex items-center bg-transparent z-10">
    <GLView
      className="w-full h-full "
      onContextCreate={async (gl) => {
        const ctx2d = new Expo2DContext(gl);
        await ctx2d.initializeText();
        setCTX(ctx2d);
      }}
    />
  </View>
  {children}
</>

); };

export default CameraView;

https://github.com/tensorflow/tfjs/assets/24165501/72247f0e-bb81-4922-9c72-1fa9667b14ef

[fazzysyed]

@gaikwadrahul8 can you please tell me whats the problem what i am doing wrong ?

[gaikwadrahul8]

Hi, @fazzysyed

Apologize for the delayed response and thank you for bringing this issue to our attention, if possible could you please help me with GitHub repo with package.json file to clone and instructions/steps to follow to replicate the same behaviour from my end ?

Thank you for your understanding and patience.

[fazzysyed]

https://github.com/Hyuto/yolov5-tfjs-react-native is a using this repo

[gaikwadrahul8]

Hi, @fazzysyed

Thank you for sharing the Github repo link, To confirm May I know what all steps you followed so I'll try to replicate the same behavior from my end ? May I know did you try web option or you just installed Expo Go app and scanned QR code ? Thank you.

[fazzysyed]

The latest version of Expo Go will be downloaded from the stores, and if the app does not run, you can connect it to the phone to run, and it will automatically download the specific version of Expo Go.

[fazzysyed]

any luck @gaikwadrahul8

[gaikwadrahul8]

Hi, @fazzysyed

Apologize for the delayed response and I was trying to replicate the same issue from my end after cloning provided Github repo and following steps which are mentioned in that Github repo but I'm getting below errors so to confirm, did you follow the exact same steps as per Github repo or different ? if you have your own Github repo where you replicate this behavior please share that Github repo with steps

1. yarn start Output

(base) gaikwadrahul-macbookpro:yolov5-tfjs-react-native gaikwadrahul$ yarn start
yarn run v1.22.21
warning ../../../../package.json: No license field
$ expo start
Starting project at /Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native
Starting Metro Bundler

› Metro waiting on exp://127.0.0.1:19000
› Scan the QR code above with Expo Go (Android) or the Camera app (iOS)

› Press a │ open Android
› Press i │ open iOS simulator
› Press w │ open web

› Press r │ reload app
› Press m │ toggle menu

› Press ? │ show all commands

Logs for your project will appear below. Press Ctrl+C to exit.
jest-haste-map: Watchman crawl failed. Retrying once with node crawler.
  Usually this happens when watchman isn't running. Create an empty `.watchmanconfig` file in your project's root folder or initialize a git or hg repository in your project.
  Error: Watchman error: std::__1::system_error: open: /Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native: Operation not permitted. Make sure watchman is running for this project. See https://facebook.github.io/watchman/docs/troubleshooting.
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: std::__1::system_error: open: /Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native: Operation not permitted
    at BunserBuf.<anonymous> (/Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native/node_modules/fb-watchman/index.js:95:23)
    at BunserBuf.emit (node:events:514:28)
    at BunserBuf.process (/Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native/node_modules/bser/index.js:292:10)
    at /Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native/node_modules/bser/index.js:247:12
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
Emitted 'error' event on WatchmanWatcher instance at:
    at Client.<anonymous> (/Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native/node_modules/jest-haste-map/build/watchers/WatchmanWatcher.js:150:10)
    at Client.emit (node:events:514:28)
    at BunserBuf.<anonymous> (/Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native/node_modules/fb-watchman/index.js:107:12)
    at BunserBuf.emit (node:events:514:28)
    at /Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native/node_modules/bser/index.js:249:12
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  watchmanResponse: {
    error: 'std::__1::system_error: open: /Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native: Operation not permitted',
    version: '2023.03.13.00'
  }
}

Node.js v18.17.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

2. expo start Output :

(base) gaikwadrahul-macbookpro:yolov5-tfjs-react-native gaikwadrahul$ expo start
Starting project at /Users/gaikwadrahul/Desktop/TFJS/test-8075/yolov5-tfjs-react-native
Error: Cannot find module 'metro-core'
Require stack:
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/server/metro/instantiateMetro.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/server/metro/MetroBundlerDevServer.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/server/DevServerManager.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/startAsync.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/index.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/bin/cli
- /usr/local/lib/node_modules/expo/bin/cli
Error: Cannot find module 'metro-core'
Require stack:
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/server/metro/instantiateMetro.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/server/metro/MetroBundlerDevServer.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/server/DevServerManager.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/startAsync.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/index.js
- /usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/bin/cli
- /usr/local/lib/node_modules/expo/bin/cli
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/expo/node_modules/@expo/cli/build/src/start/server/metro/instantiateMetro.js:10:18)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
(base) gaikwadrahul-macbookpro:yolov5-tfjs-react-native gaikwadrahul$ 

After installing the metro-core package I'm getting lots of error messages w.r.t to package dependancies even I tried yarn web option still did not work, I tried on my MacBook Pro (Apple M1 Pro chip) system

If Am I missing something here please let me know and May I know what Node.js version are you using?

Thank you for your understanding and cooperation.

[fazzysyed]

my node version v16.15.0

I am using running it on Apple M1 try this repo https://github.com/fazzysyed/Tfjs-App

[gaikwadrahul8]

Hi, @fazzysyed

Thank you for providing the information and I have observed the same behavior which you've mentioned, object detection is working slow so we'll have to dig more into this issue because this issue is related with performance

Thank you for bringing this issue to our attention, I really appreciate your valuable time and efforts. Thank you.

[alefyufca]

I'm building an application with these same functions, but when I build it with eas the text doesn't appear, but the objects are detected.

[fazzysyed]

@alefyufca you didn't face the slow detection lag? whats your model size?