Behavior of rotationX/Y/Z and accelerationX/Y/Z on mobile browsers

[LingDong-]

Hi, I would like to report some inconsistencies/failures of rotationX/Y/Z and accelerationX/Y/Z on different mobile browsers, and propose some fixes.

• A demo comparing current behavior and my patches can be found at https://p5-accelerometer-test.glitch.me/
• The full code of the above can be found at https://glitch.com/edit/#!/p5-accelerometer-test
• A video recording of the demo featuring me and 2 phones, illustrating the problem and the fix

I've tested on iOS 13.4.1 (iPhone and iPad) and Android 10 (Pixel 4) because these are the only devices I have access to.

p5.js version: 1.1.9 (latest at time of writing)

I'm not familiar with implementation details of p5, so please correct me if I've made any mistakes :)

I: iOS 13 sensor access permission

On iOS 13 webpages need to pop up a dialog to beg for the user's permission to access sensors. The dialog can only be triggered when the user has interacted with the page (ontouchstart doesn't seem to count, while ontouchend does). p5.js doesn't seem to implement this.

Proposed fix

let hasSensorPermission = !(DeviceOrientationEvent.requestPermission || DeviceMotionEvent.requestPermission);

function begPermission(){
  if (DeviceOrientationEvent.requestPermission){
    DeviceOrientationEvent.requestPermission()
    .then(response => {
      if (response == 'granted') {
        if (DeviceMotionEvent.requestPermission){
          DeviceMotionEvent.requestPermission()
          .then(response => {
            if (response == 'granted') {
              hasSensorPermission = true;
            }
          })
          .catch(alert)
        }
      }
    })
    .catch(alert)
  }
}

function touchEnded() {
  if (!hasSensorPermission){
    begPermission();
  }
}

II: accelerationX/Y/Z does not work on iOS

It seems that iOS devices (or at least all the ones I have access to) doesn't have DeviceMotionEvent.acceleration. They only have what's called DeviceMotionEvent.accelerationIncludingGravity. p5 seems to only listen to the former, and as such, the variables accelerationX/Y/Z are constantly 0.

Moreover, as its name suggests, accelerationIncludingGravity takes the gravitational acceleration into account, producing inconsistent behavior compared to devices that supports the plain acceleration.

Proposed fix

The fix involves calculating components of g=9.80665ms^-2 on each axis based on the current rotation and cancelling them out from the accelerometer reading. (Which won't work accurately in outer space or some really high parts of the Earth. navigator.geolocation.getCurrentPosition might be further used to deduce the correct g but that's probably too crazy :P).

// fixed values
let accX;
let accY;
let accZ;

// 3d transformation helpers
let ROTX = a=> [1,0,0,0, 0,cos(a),-sin(a),0, 0,sin(a),cos(a),0, 0,0,0,1]
let ROTY = a=> [cos(a),0,sin(a),0, 0,1,0,0, -sin(a),0,cos(a),0, 0,0,0,1]
let MULT = (A,B)=> [(A)[0]*(B)[0]+(A)[1]*(B)[4]+(A)[2]*(B)[8]+(A)[3]*(B)[12],(A)[0]*(B)[1]+(A)[1]*(B)[5]+(A)[2]*(B)[9]+(A)[3]*(B)[13],(A)[0]*(B)[2]+(A)[1]*(B)[6]+(A)[2]*(B)[10]+(A)[3]*(B)[14],(A)[0]*(B)[3]+(A)[1]*(B)[7]+(A)[2]*(B)[11]+(A)[3]*(B)[15],(A)[4]*(B)[0]+(A)[5]*(B)[4]+(A)[6]*(B)[8]+(A)[7]*(B)[12],(A)[4]*(B)[1]+(A)[5]*(B)[5]+(A)[6]*(B)[9]+(A)[7]*(B)[13],(A)[4]*(B)[2]+(A)[5]*(B)[6]+(A)[6]*(B)[10]+(A)[7]*(B)[14],(A)[4]*(B)[3]+(A)[5]*(B)[7]+(A)[6]*(B)[11]+(A)[7]*(B)[15],(A)[8]*(B)[0]+(A)[9]*(B)[4]+(A)[10]*(B)[8]+(A)[11]*(B)[12],(A)[8]*(B)[1]+(A)[9]*(B)[5]+(A)[10]*(B)[9]+(A)[11]*(B)[13],(A)[8]*(B)[2]+(A)[9]*(B)[6]+(A)[10]*(B)[10]+(A)[11]*(B)[14],(A)[8]*(B)[3]+(A)[9]*(B)[7]+(A)[10]*(B)[11]+(A)[11]*(B)[15],(A)[12]*(B)[0]+(A)[13]*(B)[4]+(A)[14]*(B)[8]+(A)[15]*(B)[12],(A)[12]*(B)[1]+(A)[13]*(B)[5]+(A)[14]*(B)[9]+(A)[15]*(B)[13],(A)[12]*(B)[2]+(A)[13]*(B)[6]+(A)[14]*(B)[10]+(A)[15]*(B)[14],(A)[12]*(B)[3]+(A)[13]*(B)[7]+(A)[14]*(B)[11]+(A)[15]*(B)[15]]
let TRFM = (A,v)=> [((A)[0]*(v)[0]+(A)[1]*(v)[1]+(A)[2]*(v)[2]+(A)[3])/((A)[12]*(v)[0]+(A)[13]*(v)[1]+(A)[14]*(v)[2]+(A)[15]),((A)[4]*(v)[0]+(A)[5]*(v)[1]+(A)[6]*(v)[2]+(A)[7])/((A)[12]*(v)[0]+(A)[13]*(v)[1]+(A)[14]*(v)[2]+(A)[15]),((A)[8]*(v)[0]+(A)[9]*(v)[1]+(A)[10]*(v)[2]+(A)[11])/((A)[12]*(v)[0]+(A)[13]*(v)[1]+(A)[14]*(v)[2]+(A)[15])]

window.ondevicemotion = function(event) {
  if (!event.acceleration){ // devices that don't support plain acceleration
    
    // compute gravitational acceleration's component on X Y Z axes based on gyroscope
    // g = ~ 9.80665
    let grav = TRFM(MULT(
      ROTY(radians(rotationY)),
      ROTX(radians(rotationX))
    ),[0,0,-9.80665]);
    
    accX =  (event.accelerationIncludingGravity.x+grav[0]);
    accY =  (event.accelerationIncludingGravity.y+grav[1]);
    accZ =  (event.accelerationIncludingGravity.z-grav[2]);
    
    // p5 appears to be doubling the acceleration for reasons that aren’t explained:
    // https://github.com/processing/p5.js/blob/main/src/events/acceleration.js#L647
    accX *= 2;
    accY *= 2;
    accZ *= 2;
  }
  
}

III: rotationX/Y does not take device orientation into account

When user is in landscape/portrait, rotationX and rotation Y's correspondence to rotateX() and rotateY() are swapped.

I know this is inherited from HTML's API, and the users of p5 library can always figure out the orientation of the device first and swap these variables themselves. But I think the fact that the graphics (which is forcibly rotated by the OS) is no longer in sync with the sensor data can be anti-intuitive. And as a beginner friendly library I think there's an opportunity for p5 to wrap this a bit differently (or perhaps provide it as an option for the user.)

Proposed fix

This fix involves reading window.orientation which is theoretically one of (0,90,-90,180). My phones (and probably most phones in general) don't allow people to hold them vertically upside-down (flipped portrait), so I didn't include the case.

rotX = radians([-rotationY,-rotationX,rotationY][~~(window.orientation/90)+1]);
rotY = radians([-rotationX, rotationY,rotationX][~~(window.orientation/90)+1]);
rotZ = radians(rotationZ);

IV: Documentation error on https://p5js.org/reference/#/p5/rotationX

The text says the order should be Z-X-Y (which is correct), but the example code on the same page apparently applies them in Y-X-Z order (which is incorrect).

I believe in p5.js, a series of transformations is equivalent to left matrix multiplication in reverse order, so

rotateZ(z);
rotateX(x);
rotateY(y);

is

Rz * (Rx * (Ry * v)))

Which is in fact what people call Y-X-Z order.

The correct Z-X-Y order should be written as:

rotateY(y);
rotateX(x);
rotateZ(z);

The same issue also appears in https://p5js.org/reference/#/p5/rotationY and https://p5js.org/reference/#/p5/rotationZ

[welcome[bot]]

Welcome! 👋 Thanks for opening your first issue here! And to ensure the community is able to respond to your issue, be sure to follow the issue template if you haven't already.

[limzykenneth]

Thanks for the write up.

I: iOS 13 sensor access permission

p5.js doesn't aim to handle this directly and a user side fix is relatively straightforward. We can just document this behaviour and fix instead of implementing another function that is just a thin wrapper around it.

II: accelerationX/Y/Z does not work on iOS

For me acceleration is working fine on iOS with a 1st gen iPhone SE. Do you have permission for Safari to access accelerometer data enabled in the system settings? Also according to documentation on Safari Mobile, acceleration is supported, so I would expect it to work.

III: rotationX/Y does not take device orientation into account

This is too much of a breaking change for me to recommend this fix at this point in time.

IV: Documentation error on https://p5js.org/reference/#/p5/rotationX

The lines about order of rotation is not relevant to the subject or example (since other lines of rotateX/Y/Z is commented out). These lines can just be removed entirely.

[LingDong-]

Hi, thank you for the reply.

p5.js doesn't aim to handle this directly and a user side fix is relatively straightforward. We can just document this behaviour and fix instead of implementing another function that is just a thin wrapper around it.

Sounds reasonable, but I hope the documented fix will be more sophisticated than the relatively straightforward snippet you linked to. If the device does not require asking for permission or implements it in another way than DeviceOrientationEvent.requestPermission, users will get an error by pressing the button. If you take this into account (and possibly many other things, like what if the user pressed "NO" when they're asked?) the wrapper might not be as trivial.

I always thought of p5.js as a collection of thin wrappers. Afterall there's nothing that p5 does cannot be done relatively strightforwardly with plain JS. The whole point is that users don't have to understand how the JS API's work to make cool things for the web, and in situations like these it sorts of breaks the "magic" :)

---

For me acceleration is working fine on iOS with a 1st gen iPhone SE. Do you have permission for Safari to access accelerometer data enabled in the system settings? Also according to documentation on Safari Mobile, acceleration is supported, so I would expect it to work.

On the same page you linked to, it says

This property is null if the device cannot provide the user acceleration—
that is, if the device does not have a gyroscope.

MDN says this about accelerationIncludingGravity:

This value is not typically as useful as DeviceMotionEvent.acceleration, 
but may be the only value available on devices that aren't able of removing 
gravity from the acceleration data, such as on devices that don't have a gyroscope."

I've tested on an iPhone 7 (somewhat old) and an iPad Pro (12.9-inch) (3rd generation) which I believe is fairly new. Neither of them has acceleration, which might suggest that the lack of acceleration might be not completely rare for mobile devices in general. My proposed fix above also does not target iOS specifically: it tests for all devices that does not support acceleration and falls back to accelerationIncludingGravity gracefully.

(And yes I most definately have permission for Safari, since otherwise I wouldn't be able to access rotation and accelerationIncludingGravity required in my proposed fix, or to be able to write anything about the first issue I: iOS 13 sensor access permission)

---

This is too much of a breaking change for me to recommend this fix at this point in time.

That is toatally understandable. However, it does not have to be breaking, for examples:

• add rotationMode(ABSOLUTE) and rotationMode(RELATIVE_TO_ORIENTATION). By default it's the former, which will not break existing code.
• add reorientRotation() which user can choose to call upon deviceTurned() event.

---

The lines about order of rotation is not relevant to the subject or example (since other lines of rotateX/Y/Z is commented out). These lines can just be removed entirely.

I believe the comments have an very misleading effect and not at all irrelevant. All three pages for rotationX/Y/Z mentions

The order the rotations are called is important, ie. if used together, 
it must be called in the order Z-X-Y or there might be unexpected behaviour.

AND the code shows:

//rotateZ(radians(rotationZ));
rotateX(radians(rotationX));
//rotateY(radians(rotationY));

and

//rotateZ(radians(rotationZ));
//rotateX(radians(rotationX));
rotateY(radians(rotationY));

and

rotateZ(radians(rotationZ));
//rotateX(radians(rotationX));
//rotateY(radians(rotationY));

The fact that the code consistently shows this Y-X-Z order will lead people into believing that if they would like to have all three rotations, they'll only need to uncomment the commented lines. Which will do exactly the opposite, triggering unexpected behaviour as described by the textual documentation (Which, BTW is also inaccurate, since the behavior will be expectedly incorrect).

The order of Euler angles can be a confusing subject for beginners, (it certainly seems to confuse whoever wrote these examples in the documentation). Therefore we can help them by having correct comments in the documentation.

Anyway, thanks for considering my suggestions.

[amcc]

I think one straightforward thing that would really help is to provide documentation on iOS for this. A large number of users will be using mobile safari and without permission this doesn't work. Its a shame the documentation doesn't mention anything about this.

I teach students with p5js regularly and the steps to get round iOS permission are probably too complex to expect most early javascript coders to understand. An example, or a p5 method that wraps this up would be nice.

Here's a sketch I made that works on iOS - its very similar to the solution by @LingDong- above. https://editor.p5js.org/amcc/sketches/kBndhSZER

It seems that this is a very similar scenario to https://p5js.org/reference/#/p5/userStartAudio which is provided for mobile sound interactions.