Mobile integration

[rkz98]

Is there any mobile integration for AR stuff? (React-Native, Kotlin or Swift)

[hungtooc]

I've tried to run onnx model on react native, it's still quite slow

[rkz98]

@hungtooc may you share the implementation code? I'd like to try it out.

[hungtooc]

@rkz98 here is react native code:

import { InferenceSession, Tensor, } from "onnxruntime-react-native";
// const IMAGE_PATH = "/assets/data/beach.jpg";
const IMAGE_EMBEDDING = "/assets/data/beach_embedding.npy";

const App = () => {
  const [model, setModel] = useState<InferenceSession | null>(null); // ONNX model
  const [tensor, setTensor] = useState<Tensor | null>(null); // Image embedding tensor

  useEffect(() => {
    // Initialize the ONNX model
    const initModel = async () => {
      try {
        const model = await InferenceSession.create('file://' + RNFS.DocumentDirectoryPath + '/sam.onnx');
        console.info(model.inputNames)
        setModel(model);
      } catch (e) {
        console.log(e);
      }
    };
    initModel();

    // Load the Segment Anything pre-computed embedding
    Promise.resolve(loadNpyTensor(IMAGE_EMBEDDING, "float32")).then((embedding) => {
      console.info("embedding.dims", embedding.dims)
      setTensor(embedding)
    });
  }, []);
const loadNpyTensor = async (tensorFile: string, dType: string) => {
    // notice I created fake data
    const data = new Float32Array(256 * 64 * 64);
    const tensor = new Tensor("float32", data, [1, 256, 64, 64]);
    return tensor;
  };
## end of init

useEffect(() => {
    console.info("runONNX once")
    if (model && tensor)
      runONNX(model, [{
        clickType: 1,
        x: 150,
        y: 150
      }], tensor, {
        samScale: 1.28,
        height: 640,
        width: 480
      });
  }, [state]);

}

[eikaramba]

any way to also generate the embeddings on mobile?

[creativekdev]

I am trying to integrate it on flutter plugin. is it possible?

[ryouchinsa]

You can find the converter for the encoding part to the onnx format in the repository Segment Anything CPP Wrapper.

Here is the Segment Anything CPP Wrapper for macOS. This code is originated from Segment Anything CPP Wrapper and implemented on macOS app RectLabel. We customized the original code so that getMask() uses the previous mask result called as low_res_logits and retain the previous mask array for undo/redo actions.

We hope this macOS version would help to develop the iOS version. Please let us know your opinion.

[ryouchinsa]

Improved "Create polygon using SAM" feature so that you can label pixels using the pixels option. You can label Segment Anything 1 Billion (SA-1B) like dataset by yourself.