Reimplementation of device orientation code - iOS/Safari user participation needed!

[nickw1]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Rotation of AR view is often 90 degrees out on Firefox (but not consistently).

If the current behavior is a bug, please provide the steps to reproduce.

Try a location-based example, such as https://hikar.org/webapp/ (in Europe and Turkey), on Firefox, you will find that the augmented content might be 90 degrees out.

Investigating this issue

I'd like to investigate this issue a bit further, to find out whether it is within AR.js or whether it is in the THREE.js DeviceOrientationControls used by A-Frame to perform automatic rotation based on the compass and accelerometer sensors.

To do this I have created a very simple pure A-Frame (no ar.js) example here: https://hikar.org/aframe/

Please try this on your own device, in a location where you will not get magnetic interference, and see if you get the expected behaviour, which is:

• Red sphere to your north
• Green sphere to your east
• Yellow sphere to your south
• Blue sphere to your west

Please report the behaviour on your device, stating device, operating system and critically, browser and version. Note on Firefox you have to tap the screen to start the auto-rotation.

If this works for most devices, we know the problem is in AR.js. If it frequently fails on Firefox, we know the problem may well be with DeviceOrientationControls.

I will start by reporting ~~expected behaviour~~ unexpected behaviour: sometimes the spheres are 180 degrees out on

• Chrome 91, Android 11, Pixel 3
• Firefox 89, Android 11, Pixel 3

Thanks. The more reports we get, the more we can be sure whether the issue is likely to be within AR.js or not.

[nickw1]

Just to add to this, there are three functions in gps-camera and gps-projected-camera which I am not sure the reason for, and which might possibly be causing problems:

_onDeviceOrientation()
_computeCompassHeading()
_updateRotation()

They seem to be doing some kind of update of the yaw dependent on the rotation:

this.lookControls.yawObject.rotation.y = THREE.Math.degToRad(offset);

but given that the three.js DeviceOrientationControls, included in A-Frame, already handles this, are they actually needed and might they be introducing problems?

@nicolocarpignoli any idea on this?

[nickw1]

Another update:

I tried removing the three methods in the last post; they are necessary as the augmented content is completely out of sync with the real world if they are removed.

Also, and importantly, I have found that the basic A-Frame example on the first post in fact does not work consistently on my devices, either under Chrome or Firefox. Sometimes it's 180 degrees rotated so that the 'north' sphere appears to the south. Refreshing the page sometimes fixes it.

In conclusion, as the basic A-Frame example does not reliably work, I'm wondering whether there is a bug in the version of THREE.DeviceOrientationControls being used, as a plain A-Frame example is not detecting north, south, east and west correctly. I am using A-Frame 1.0.4 though, will upgrade to 1.2.0 and see if that makes a difference.

[nickw1]

Starting to get to the bottom of this I think.

The problem is that we cannot rely on the alpha property of DeviceOrientationEvent - see here - actually returning a value relative to (magnetic) north. If the absolute value is false - see here, then, to quote, "data is being provided in reference to some arbitrary, device-determined coordinate frame" - without specifying what that arbitrary frame is.

In tests, I am finding that absolute is false, unfortunately, on both Chrome and Firefox on Android.

However, alpha gives the expected value - rotation relative to north - on Chrome/Android. On Firefox/Android, it sometimes does, and sometimes doesn't. Sometimes it's about 90 degrees out, sometimes 180. This accords exactly with problems people have been reporting on Firefox and is discussed further below.

This (admittedly not up-to-date) article on W3C suggests that absolute ought to be true on Firefox/Android and Chrome/Android, but it isn't, suggesting that it's changed in recent years. However it does suggest that alpha is relative to the initial bearing of the device when the device orientation is first tested for, which at least gives us something to work with. I am also finding on Firefox that the A-Frame sphere example mentioned above always shows the red sphere in front of you initially, whether you're facing north or not - which would tie in with this.

Try out an alpha/beta/gamma demo on your device here if you're interested.

[nickw1]

The fun continues...

Looks like we can solve this reliably on Chrome, and also possibly Safari, using the deviceorientationabsolute event, and webkitCompassHeading respectively. Though strangely, Chrome seems to work anyway - perhaps that's what the three methods I mentioned in the second post above do.

Can't find a solution for Firefox to be honest. Guessing we might have to take the raw accelerometer and magnetic field values, and calculate an orientation matrix from them, as has been done in the Android API. Maybe someone has developed a library for that already - need to investigate.

Or - less good - calculate the heading from successive GPS lat/lon readings from the Geolocation API, obviously this is no good if you're standing still though.

If anyone has any thoughts on this, do please post them here.

Stack Overflow discussion here

[nickw1]

I'm wondering whether, sadly, it is currently impossible to do this on Firefox.

It seems that the Accelerometer, Magnetometer and AbsoluteOrientationSensor APIs are not available on Firefox.

https://developer.mozilla.org/en-US/docs/Web/API/Magnetometer https://developer.mozilla.org/en-US/docs/Web/API/Accelerometer https://developer.mozilla.org/en-US/docs/Web/API/AbsoluteOrientationSensor

What seems a little inexplicable is that apparently this used to be possible as the DeviceOrientationEvent, according to the W3C document above, formerly gave an absolute value.

If anyone can suggest a way of getting this working on Firefox in the meantime, that would be great - otherwise for now, I think we're stuck with limiting support to Chrome and Safari.

[nickw1]

OK I have found some time to start investigating this and to replace the current device orientation code with a new implementation which takes into account changes in the device orientation API.

I have a small demo at https://hikar.org/orient2/ which shows the current azimuth (compass bearing), pitch and roll of the device.

This gives the expected values on Chrome, in portrait (EDIT landscape is more complex - this is a todo)

It will not work on Firefox; as said above, sadly there appears to be no way to currently obtain absolute orientation on Firefox.

It should in theory work on Safari but I have no way of testing this. Therefore, if anyone with an iOS device is able to test out the demo on Safari and report how:

• the azimuth changes as you change the compass direction that the device is heading, in portrait;
• the pitch and roll change as you flip the device up and down and roll it from side to side, in portrait.

... that would be great.

Thanks, Nick

[cameronValtech]

@nickw1 At the moment, on Safari/iOS 14, I get an alert "aaa", and after that a page with no orientation data.

[Sheeban-Wasi]

Hey, this discussion seems relevant to my problem can someone help? There is a 3D photo hosted on the link whenever someone open the link on his phone and give the orientation permissions he can move the phone or rotate it and see different views of the 3D picture inside his phone. Hope I make sense.

[nickw1]

@nickw1 At the moment, on Safari/iOS 14, I get an alert "aaa", and after that a page with no orientation data.

@cameronValtech ok thanks. This is proving to be a difficult one. Are you able to run it with the developer tools for Safari? Would be good to see if there is a specific error. No worries if not!

The 'aaa' is just a debug thing btw...

[ThorstenBux]

@nickw1 , this is the error message on iOS 15:

No device orientation events will be fired, reason: Permission to use the API was not yet requested.

You would need to add a button which actually requests access to the API

[nickw1]

@ThorstenBux ok thanks.

[drewish]

I was able to use the demo on iOS 15.1 and it seemed to be working correctly. Azimuth seemed to correspond to the compass heading (when the device laying flat on a table or held upright), pitch to corresponded to the device's orientation laying flat or standing upright, and roll to the side to side rotation that would shift between portrait and landscape. I couldn't figure out what heading corresponded to though.

[nickw1]

@drewish many thanks. Did azimuth correspond to the compass heading when the device was NOT lying flat?

heading is calculated from some code present in AR.js, written by @nicolocarpignoli , just investigating if this is more accurate than the azimuth.

[drewish]

Heading seemed to have no relation to the compass. It would also jitter erratically by a 2 or 3 when the phone was laying stationary on the table