google-ar
Issue with tracking lost - ML based IMU?
Hi!
I have a question regarding the ML- base IMU tacking mentioned in last year's Google IO (2022).
In the presentation “A deep dive into ARCore” it is explained how the Visual-Inertial Sensor Fusion can be used to correct tracking between frames. However, if there is too much of a gap in between the frames the tracking will stop. It is then explained how a new ML based method can be used to update the player position during longer “blackouts”. So, will this ML based IMU be released to AR Core? I can’t see any updates here mentioning its release.
Using Unity ARFoundation, I can show an example on how easy it is to lose tracking using ARCore (Left) and the consequence that it causes to the users experience. For comparison I have the same test scenario using ARKit (Right).
https://github.com/google-ar/arcore-android-sdk/assets/115237780/4d1bbe0b-8ed0-4e65-8fb6-dc0497d0f53f
Video explanation: I'm blocking the camera with my hand and rotate it 90 degrees to the left. We can see that the tracking completely stops on ARCore once the camera is blocked. After removing the hand, the AR scene has moved 90 degrees to the left. Rotating back, the AR scene jumps back to the correct position. Doing one more short block of the camera, and only applying a small circular motions (as seen in ARKit version), the system once again incorrectly rotates the AR scene 90 degrees to the left.
Unfortunately, this lack of tracking capabilities and issue handling during blackouts makes it difficult to create an production- ready Android/ARCore product.
As promised at the end of the "A deep dive into ARCore" presentation, the issue is fixed in Google Maps Live View, and works well using the same Android hardware and camera blockage using my hand.
Not sure if;
- I have missed something when configuring ARFoundation/ARCore plug-in
- There is a bug in ARCore, causing the tracking loss/freeze
- The ML IMU is not released on ARCore, but will be released
- The ML IMU will not be released on ARCore
Otherwise, does anyone have a proposal on how to prevent the tracking from freezing or handle the issue when it occurs? I have already tested calculating the double integral of the acceleration with extreme fluctuations as results, as expected.
Using: Unity 2020.3.17f1 AR Foundation 4.2.7
Does not work on:
- Samsung Galaxy S10 5G - Android version 12 - Google play services for AR: 1.37.23.1000493
- Android Pixel - Android version 10 - Google play services for AR: 1.36.230390423
Works properly on: -
Thanks!
@devbridie Any news on that? This is making ARCore unusable in a lot of cases that are well handled by ARKit. Even more since the coordinate system changes when camera tracking is retrieved. Making us having to retrieve anchors poses to get their position in the new coordinate system. Worst is that if the anchor is not in the Camera View but its renderable is, it will always appear wrongly placed (not trackable anchor in the old coordinate system)
Totally agree, it's really hard to recover (in a reliable way) after tracking is lost with ARCore. You will probably end up with a "world discontinuity" and there is really no other reliable way to recover than resetting the session.
@devbridie Any comments or reflection on this would be highly appreciated. Today this issue creates problems in our applications and it would create a peace of mind if we could get a hint on future updates that could tackle this or make it better?
@devbridie I'd also be interested in hearing an update on this, for the same reasons as others on the thread
@devbridie I also noticed this is happening a lot - esp in scenarios where there are not as much features. ARKit works in these cases.
