Disruption of video frame order related to tf.browser.fromPixels(HTMLVideoElement) in recent Firefox

[eyaler]

System information

• I written custom code
• Windows, issue on Firefox 94-95 compared to older versions of FF + chorme/edge)
• PC
• TensorFlow.js from jsdelivr.net/npm tried versions from 2.4.0 to 3.12.0

Describe the current behavior Running mediapipe facemesh estimateFaces on videoElement in Firefox 94/95 causes the video to be extremely jerky with frames jumping back and forth in time. I think this is related to tf.browser.fromPixels(HTMLVideoElement). It was OK in previous versions of FF (not sure which but it was OK a few months ago), and it is OK on chorme/edge and it is also OK if I use a camera stream instead of video. Link to demo showing the issue below (wait for the model loading to complete + you can hide/remove the mask to see that the issue is with the underlying video itself)

Describe the expected behavior the video frames should be displayed in order without jumping back and forth.

Standalone code to reproduce the issue demo: https://eyaler.github.io/tfjs_webgl_app/face_landmark/ this is the line causing the issue: https://github.com/eyaler/tfjs_webgl_app/blob/master/face_landmark/webgl_main.js#L650

thanks!

[eyaler]

@lina128 any news on this please?

[rthadur]

Looks like issue is not reproducible in latest version of Mozilla , can you please check. Thank you

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[eyaler]

@google-ml-butler, this issue is open since december 2021, so i do not appreciate the pressure to mark this as stale/closed after just a few days from last call to action.

@rthadur, thanks. i just checked on firefox 105.0.1 (64-bit) on windows 11, with tfjs 3.12.0 and i see the same issue. i did not try with later tfjs versions. did you test with a later version? is there a commit addressing the issue?

[rthadur]

We do not have windows machine , I am not sure if there is a fix , please try with latest version and update the issue , thank you

[eyaler]

I updated both tfjs and tf-backend-wasm to the latest 3.20.0 and seeing the exact same issue :(

[eyaler]

any news please? @rthadur @lina128

[gaikwadrahul8]

Hi, @eyaler

Apologize for the delayed response and we're re-visiting our older issues and checking whether those issues got resolved or not as of now so May I know are you still looking for the solution or your issue got resolved ?

If issue still persists after trying with latest version of @[email protected] and @tensorflow/[email protected] please let us know and I tried on my Mac M1 system and it's working as expected and I don't have windows system to test it from my end now so I would request you to please try it from your end with latest versions and let us know whether are you still facing the same issue or not ?

Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ? Thank you!

[eyaler]

@gaikwadrahul8 thanks for your kind followup. i can confirm that the issue was solved when going from firefox 107.0.1 (broken) to 108.0 (fixed; dec 2022) i tested with both tfjs 3.20.0 and 4.6.0 i believe the next step should be to figure out what was the relevant changeset and understand any other possible implications. i hope you will not blindly close this bug as this was a breaking issue with video playback that prevented using tfjs for video on firefox for ~1.5 years. bugs that magically disappear always call for a deeper investigation, important lessons can be learned.

[gaikwadrahul8]

Hi, @eyaler

Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.

The TFJs team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TFJs version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.

Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow.js space.

Thank you for your support and cooperation.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No