Vision / Tracking - parity with ARFoundation

[xwipeoutx]

Describe the problem

iOS and Android have easy ways to "align the world" - object tracking, face tracking, image markers, QR codes. It's all easy.

I can't do this easily on Microsoft devices like the surface pro or HoloLens, without a 3rd party (eg. Vuforia)

Describe the solution you'd like

I would like device APIs (ideally) or MRTK implementations to do all the things that I can do in my Android and iOS tablet available on Surface Pro-type devices, and rich access to functionality exposed in ARFoundation across these and HoloLenses. Including

• Image Tracking
• Object Tracking
• Face detection / skinning
• AR Support (for devices running windows eg. Surface Pro - esp. 2 in 1)
• QR Code

Ideally these would be accessible via ARFoundation.

Describe alternatives you've considered

I've used Vuforia to cover some of this, but it's clearly dated - compare ARCore to Vuforia for image tracking

Additional context

Mostly, I want to be able to take the code I've already done for ARFoundation, and have it Just Work on 2-in-1 laptops, and the HoloLens

[handyjellyfish]

The 2 in 1 surface devices are potentially great for this sort of work, and it's really unfortunate that MS is lagging in this space that Apple and Google have smothered. Vuforia are the only providers of an AR solution on MS surface devices, and it is lacking. It would be fantastic to see Microsoft work towards feature parity in this area.

Definite +1 request from me.

[david-c-kline]

Thank you for the feedback @xwipeoutx!

[gilbdev]

definitely one of the must-have features when one's want to do some AR stuff! +1

[genereddick]

+1

I would add scene / camera scaling to the list

[anders-lundgren]

Great suggestion! I think these are super important features that need to be added ASAP, both to the Surface line of devices and to the HoloLens and/or MRTK. Aside from being extremely useful, it would signal that MS commitment to Mixed Reality is strong and ongoing. From what I gather, it should be fairly straightforward to get a decent implementation in place on the Surface line, since there is Augmented/Mixed Reality support in the 3D Viewer for Windows 10, that would make perfect sense to improve by adding marker support and opening up for third party applications. For the HoloLens, even a port of the NFT feature from the open source toolkit ARToolkit would be helpful to include in MRTK, see https://github.com/qian256/HoloLensARToolKit and https://github.com/artoolkit/ARToolKit5

In the long run, a roadmap that includes full parity compared to competing frameworks, including facial tracking, object recognition, people occlusion, is also something that is needed to keep traction and interest from developers and investors.

[Fangh]

Moreover Unity3D team has written in the last blog post about 2019.3 that ARFoundation is now supported on the HoloLens. So maybe you should contact them and tell them that this is not the case. Because that's giving false hope...

[xwipeoutx]

Hey @davidkline-ms , how this is looking from a planning / prioritisation perspective? Has there been any chatter on the product team for some of these ARFoundation features (image/face/object tracking) to be device-native?

[david-c-kline]

sorry about the delay in seeing this!

i am actively looking at the XR Interaction toolkit to see how MRTK and it may work together in the future. i hope to have more to share in the not too distant future.

again, sorry i missed seeing this for so long

[Zee2]

We're using ARFoundation for scene reconstruction/meshing across all platforms in MRTK3! I'll keep this issue active to track an overall community desire for easy-to-use world-locking tools.

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!