Model runtime takes longer than expected

[arianaa30]

Following the online tutorial for selfie segmentation, I made this simple code using latest API version (com.google.mlkit:segmentation-selfie:16.0.0-beta4). Do we know how long it takes to segment, say a low-res 300x250 image on a high-end phone (Samsung S22)?

When I time the model processing like below, it shows almost 100ms only for the model to return the result. Let alone the mask processing, etc. Am I missing anything? Is there a way to optimize it further?

    SelfieSegmenterOptions options =
            new SelfieSegmenterOptions.Builder()
                    .setDetectorMode(SelfieSegmenterOptions.STREAM_MODE)
                    .build();
    Segmenter segmenter;
    segmenter = Segmentation.getClient(options);


    InputImage image;
        image = InputImage.fromBitmap(imageBitmap, Surface.ROTATION_0);

    long startTime = System.nanoTime();

    Task<SegmentationMask> result =
            segmenter.process(image)
                    .addOnSuccessListener(
                            new OnSuccessListener<SegmentationMask>() {
                                @Override
                                public void onSuccess(SegmentationMask segmentationMask) {
                                    // Task completed successfully

                                    long endTime = System.nanoTime();
                                    String time = "Time: "+(endTime - startTime)/1000000f +" ms";
                                    timeTextView.setText(time);

                                    ByteBuffer mask = segmentationMask.getBuffer();
                                    int maskWidth = segmentationMask.getWidth();
                                    int maskHeight = segmentationMask.getHeight();

                                    // the rest of code...mask processing and making it pink
                                }
                            })
                    .addOnFailureListener(
                            new OnFailureListener() {
                                @Override
                                public void onFailure(@NonNull Exception e) {
                                    // Task failed with an exception
                                }
                            });
}

[SilverDestiny]

Hi @arianaa30 ,

Is the input static image/single image? If yes, you need to use SelfieSegmenterOptions.SINGLE_IMAGE_MODE instead of SelfieSegmenterOptions.STREAM_MODE.

I tested by https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart amd on Pixel 3 device with 1440x1080 input resolution, the inference is around ~30ms.

Please note that the 1st inference is higher than the following inferences because of initialization task and OS scheduling. Streaming mode's latency might be stable, you can also check it from the streaming activities in https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart

[arianaa30]

Hmm is there anything weird with the above script that makes it slower on a higher-end phone? It's doing the same thing as vision-quikstart. It's from your online tutorial. In my case, I have to pass every frame of video as media.Image to this module to process, and get the results back. So it's considered a STREAM mode eventually.

[SilverDestiny]

I didn't see any issue in the code snippet you provided above.

Probably one thing to note is that please make sure the result task runs immediately after it's created. If there're other pending tasks, the calculated latency might be high as the result task might be waiting to be run

[arianaa30]

@SilverDestiny per your suggestion, in my code snippet above, I'm running the result task immediately after it's created right? Is this what you mean?

    Task<SegmentationMask> result =
            segmenter.process(image)
                    .addOnSuccessListener(
                            new OnSuccessListener<SegmentationMask>() {