tfjs icon indicating copy to clipboard operation
tfjs copied to clipboard

Blazeface result is not correct on WASM backend

Open deepanshusharma012 opened this issue 3 years ago • 19 comments

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): Yes, I am using the estimateFace function of blazeface to detect the actual face in the image uploaded. Here is my code for reference:
  const detectFace = async () => {
    const p = document.getElementById('result');

    p.innerHTML = "Detecting ...";
    console.log('Loading model...')
    const model = await blazeface.load({scoreThreshold: 0.9});
    console.log('Loaded model...')

    var predictions;
    var t0, t1;

    for (j = 0; j < 5; j++) {
        var t0 = performance.now()
        // Pass in a video stream (or an image, canvas, or 3D tensor) to obtain an
        // array of detected faces from the MediaPipe graph. If passing in a video
        // stream, a single prediction per frame will be returned.
        predictions = await model.estimateFaces(canvas, false);
        console.log(predictions);
        var t1 = performance.now()
    }
    if (predictions.length > 0) {
        var prediction = predictions[0]
        var text = "The face detection completed in " + (t1 - t0) + " ms.<br>The face matched with predictions of " + predictions[0].probability[0];
        text += "<br>"
        text += `<strong>Right Eye:</strong> <br> x = ${prediction.landmarks[0][0]}; y = ${prediction.landmarks[0][1]}<br>`
        text += `<strong>Left Eye:</strong> <br> x = ${prediction.landmarks[1][0]}; y = ${prediction.landmarks[1][1]}<br>`
        text += `<strong>Nose:</strong> <br> x = ${prediction.landmarks[2][0]}; y = ${prediction.landmarks[2][1]}<br>`
        text += `<strong>Mouth:</strong> <br> x = ${prediction.landmarks[3][0]}; y = ${prediction.landmarks[3][1]}<br>`
        text += `<strong>Right Ear:</strong> <br> x = ${prediction.landmarks[4][0]}; y = ${prediction.landmarks[4][1]}<br>`
        text += `<strong>Left Ear:</strong> <br> x = ${prediction.landmarks[5][0]}; y = ${prediction.landmarks[5][1]}<br>`
        renderPrediction(predictions);
    } else {
        var text = "No face found in the image.";
    }
    p.innerHTML = text;
}
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): This behaviour is reproducible across multiple OS of both desktop & mobile devices.

  • TensorFlow.js installed from (npm or script link): https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]

  • TensorFlow.js version (use command below): 3.1.0

  • Browser version: this behavior is reproducible across multiple Browser versions including latest Chrome 92

Describe the current behavior I am using the estimateFace function of blazeface to detect the actual face in the image uploaded. It is giving 0.9804430603981018 probability when I placed the mobile device image to detect. If the face is present in the frame, then it detects properly the face and results are as expected. But if we don't have any face in the frame, in that case the blazeface detection results shows that face exists while it is Mobile device with 0.9804430603981018 probability with 0.9 scoreThreshold. (Attaching the image for reference)

Describe the expected behavior It should show result of face detected only if any face is present in frame. If there is no face in frame, then there is no point to show face detected with such high probability i.e. of 98% for mobile device on frame.

Standalone code to reproduce the issue You can use the following codepen link to reproduce: https://codepen.io/deepanshusharma012/pen/MWmRaLW Attaching the mobile device pic on which the issue is reproducible, Also the blazeface models that I'm using for face detection.

Issue Reproducible Snapshot: Screenshot from 2021-08-14 17-17-12

Image to reproduce the issue Proctoring Snapshot - 2021-08-12T11_03_56 388Z

Blazeface models used in face detection: blazeface models.zip

deepanshusharma012 avatar Aug 14 '21 12:08 deepanshusharma012

@lina128 @rthadur Hope you are doing great! Is there any update on this issue or any ETA to resolve it? Actually this issue is a blocker for our application and detecting the false events. It would be great if you respond on it and try to roll it's fix asap.

deepanshusharma012 avatar Sep 15 '21 12:09 deepanshusharma012

Hi @deepanshusharma012 , thank you for reporting this. I tested it with our own demo, same problem. Actually we're working on a new BlazeFace model, which is going to be more accurate. It will be released this quarter.

lina128 avatar Sep 15 '21 21:09 lina128

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

google-ml-butler[bot] avatar Sep 30 '21 14:09 google-ml-butler[bot]

The work is still going on this issue. I hope we'll be getting activity on this issue, so no need to mark it stalled.

deepanshusharma012 avatar Sep 30 '21 15:09 deepanshusharma012

Hi @lina128, as you said the new BlazeFace model (more accurate one) will be released by Q3 quarter. So, are we still working on it, or it has released? or any ETA for the same.

deepanshusharma012 avatar Oct 05 '21 04:10 deepanshusharma012

Sorry, I actually meant Q4. I replied in September, and we are already thinking for Q4 :) Sorry for the confusion.

lina128 avatar Oct 05 '21 23:10 lina128

@lina128 Is there any update on the new BlazeFace model?

deepanshusharma012 avatar Jan 31 '22 09:01 deepanshusharma012

Added @ahmedsabie who is working on it.

lina128 avatar Jan 31 '22 18:01 lina128

@lina128 Thanks for the update.

@ahmedsabie Do we have any ETA to get the new BlazeFace model (more accurate one) released?

deepanshusharma012 avatar Feb 02 '22 16:02 deepanshusharma012

hey @deepanshusharma012 we are aiming for sometime this month

ahmedsabie avatar Feb 04 '22 22:02 ahmedsabie

@deepanshusharma012 did you get chance to check latest blazeface model here https://github.com/tensorflow/tfjs-models/blob/master/face-detection , it is now deployed.

rthadur avatar May 03 '22 17:05 rthadur

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

google-ml-butler[bot] avatar May 10 '22 18:05 google-ml-butler[bot]

@rthadur will test shortly and share results.

ayadav avatar May 14 '22 12:05 ayadav

Closing this issue , please @mention to reopen, if the issue still persists or open a new issue with latest information. Thank you

rthadur avatar May 19 '22 16:05 rthadur

Are you satisfied with the resolution of your issue? Yes No

google-ml-butler[bot] avatar May 19 '22 16:05 google-ml-butler[bot]

Hi @rthadur , the above issue still persists for latest blazeface models having modelType set as 'short' (2 metres). Will there be any updates for the same? It is currently working fine for model type set as 'full'(5 metres). Image for which the issue persists: Testing Image

rahulgarg03 avatar Jul 27 '22 10:07 rahulgarg03

We have deprecated blazeface API, please use face-detection instead: https://github.com/tensorflow/tfjs-models/tree/master/face-detection

lina128 avatar Aug 06 '22 00:08 lina128

Hi @lina128, I have used the face-detection model. But in this face-detection model for modelType set as 'short' (2 metres) the issue still persists. Will there be any updates for the same?

rahulgarg03 avatar Aug 08 '22 12:08 rahulgarg03

Thank you for testing out the new API, I will investigate.

lina128 avatar Aug 09 '22 00:08 lina128

Hi @lina128 Is there any update on the above issue?

rahulgarg03 avatar Oct 21 '22 07:10 rahulgarg03

Hi, @deepanshusharma012

Apologize for the delayed response and we're re-visiting our older issues and checking whether those issues got resolved or not as of now so May I know are you still looking for the solution or your issue got resolved ?

If issue still persists after trying with latest version of TFJs and please refer this official documentation please let us know with error log and code snippet to replicate the same issue from our end ?

Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ? Thank you!

gaikwadrahul8 avatar May 22 '23 20:05 gaikwadrahul8

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

google-ml-butler[bot] avatar May 29 '23 20:05 google-ml-butler[bot]

Hi @gaikwadrahul8 , For our current use case we are majory targetting using this for webcam and front facing cameras on smartphone, so as per the documentation model type should be set as short. [https://drive.google.com/file/d/1d4-xJP9PVzOvMBDgIjz6NhvpnlG9_i0S/preview]

I have tested the same with the latest face detection models for above image. It is giving the same issues when tested for modelType as short. In case we set the model type as full it works correctly.

SarthakBhatia2018 avatar Jun 02 '23 17:06 SarthakBhatia2018

Hi, @SarthakBhatia2018

Apologize for the delayed response and I hope you're aware of tfjs-models/blazeface is deprecated in favor of the new face-detection package as per the official documentation so please try to use new face-detection package if issue still persists with new face-detection then please let us know with error log or with output of image. Thank you!

gaikwadrahul8 avatar Jun 05 '23 15:06 gaikwadrahul8

Hi @gaikwadrahul8,

Checked the same with mediapipe-gpu, tfjs-webgl and tfjs-wasm backends. Attaching the results with images :

with mediapipe-gpu 2

1

with tfjs-wasm :- 6

5

with tfjs-webgl 4

3

Let me know if any addtional information is required.

SarthakBhatia2018 avatar Jun 09 '23 12:06 SarthakBhatia2018

Hi @gaikwadrahul8 let me know if there is any update on above.

SarthakBhatia2018 avatar Jun 15 '23 04:06 SarthakBhatia2018

Hi, @SarthakBhatia2018

Apologize for the delayed response and could you please help us with your codepen example which you tried with latest version of @tensorflow/[email protected] or previous stable version of it and @tensorflow-models/face-detection so I'll try to replicate the same issue from my end ? Thank you!

gaikwadrahul8 avatar Jun 21 '23 08:06 gaikwadrahul8

Hi @gaikwadrahul8, I have added the code for testing face detection by uploading images. You can clone the following repository repo-link and switch to upload_image brach. Navigate to face-detection -> demos -> upload_image. folder-link

The steps to run it are the same as other folders in demos. Refer demo

Let me know if anything else is required from my end.

SarthakBhatia2018 avatar Jun 22 '23 12:06 SarthakBhatia2018

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

github-actions[bot] avatar Jun 30 '23 02:06 github-actions[bot]

Hi @gaikwadrahul8 let me know if there is any update on the same.

SarthakBhatia2018 avatar Jun 30 '23 02:06 SarthakBhatia2018