immersive-web
Choosing cameras in video mixed AR
If we're going to provide a simple video-mixed AR set of capabilities soon, how are we going to let programmers choose which camera on a multi camera device?
For example, if I want to make a "try on the sunglasses on my AR store page" AR app, I need to select the front facing camera, and either do simple computer vision or access native face detection / tracking. We've talked about ideas for the latter (CV / native apis) but it's not clear how to deal with the camera.
Is this part of session creation? Some that can be changed after the session is started?
Looking at ARKit, it would need to be part of session creation, for example, but that could be hidden internally (i.e., you switch cameras, ARKit is restarted with a new session using the appropriate camera, existing anchors are likely destroyed, etc). So, we really can choose ...
We were thinking of prototyping this in the WebXR Viewer (adding ARKit Face tracking on the front-facing camera on iOS), so I was curious what people thought was a reasonable way.
This seems like something that should definitely be part of the discussion that @NellWaliczek started during the f2f about the general session capabilities syntax.
@TrevorFSmith can we migrate this to proposals, per the discussion at the F2F
ARCore is still missing camera changing support. https://github.com/google-ar/arcore-android-sdk/issues/128
FWIW, ARCore 1.7 added support for front-facing cameras:
https://github.com/google-ar/arcore-android-sdk/releases/tag/v1.7.0 https://developers.googleblog.com/2019/02/new-ui-tools-and-richer-creative-canvas.html
However, I think there's more to this issue than just a simple camera selection. It seems that ARCore's front-facing camera mode is largely intended for use with its new face mesh type, and the release notes say that Motion tracking, all types of anchors, Augmented Images, and plane detection are not available when using the front-facing camera.
If I'm understanding it right, no motion tracking would mean no poses, so the application would basically need to use viewer space, and the XR session wouldn't really be able to provide any useful data to the application unless it exposes a face mesh or raw camera images.
@klausw that's pretty much the same as ARKit; when using the user-facing camera, all you get is faces. You might get orientation, even though there is no position.
The way I handled it when I implemented it in the WebXR viewer is to just set the pose position to 0/0/0, and feed back the fact targets relative to it.
I'd agree that until we have face mesh support, there's no particular reason to support this.
I was thinking about this recently, and I'm starting to feel like AR using the front-facing camera might actually qualify for a separate session mode? Which I know sounds weird given that we just spent a lot of time resisting separate modes for handheld/headset AR, but the use cases here are significantly different. It's hard to imagine a front-facing camera usage that could at all gracefully fall back to AR headset usage. (Although facial detection features should be available to either.) Similarly, most sites aren't going to want to advertise the availability of selfie filters or similar if all the device is capable of is rear-facing AR. Also, it seems like another instance where a early determination of the runtime/init args may be necessary (similar to Daydream/AR Core). Finally, if it was a separate mode it would be easier to spec that it only required viewer ref space support.
