Touch events report coordinates relative to the phone screen itself in browser builds

[dearlordylord]

Bevy version

0.10.1

What you did

Compiled into wasm / attached to a specific canvas (but I think no explicit canvas will do as well)

What went wrong

• what were you expecting?

touch events are relative to the canvas

• what actually happened?

touch events are reported relative to the screen (not even to the browser window!)

Other information that can be used to further reproduce or isolate the problem.

• screencast

https://github.com/bevyengine/bevy/assets/947595/835a338b-028b-436c-9d42-f258e221b68f

• workarounds that you used

explicitly send canvas.getBoundingClientRect() and use this data in your system to calculate the actual touch coords

• links to related bugs, PRs or discussions

probably https://github.com/bevyengine/bevy/issues/7528

• reproduction case deployed

https://touchies.apps.loskutoff.com/

• reproduction source code

https://github.com/Firfi/touchies

• notes

please note that the touch is being reported not even related to the browser window (which I could've worked around giving the app its canvas coordinates) bit the phone screen itself, as shown in the attached screencast

for mouse event comparison, the example deployed also has mouse event system; click couple times (to focus the element first, then to emit the actuan event)

[dearlordylord]

workaround would be:

static mut outside_pos: Option<Vec2> = None;

#[wasm_bindgen]
pub fn report_canvas_screen_position(x: f32, y: f32) {
  let v = Vec2::new(x, y);
  unsafe {
    outside_pos = Some(v);
  }
}

fn outside_window_size_system(mut event_writer: EventWriter<OutsideWindowResize>, mut size: ResMut<OutsideWindowSize>) {
  unsafe {
    if let Some(op) = outside_pos {
      if op != size.0 {
        size.0 = op;
        event_writer.send(OutsideWindowResize(op));
      }
    }
  }
}

// then in ts, call report_canvas_screen_position with the results of `canvas.getBoundingClientRect()`

// I.e. React hook:

const useCanvasPositionRelativeToTheWholeScreen = (canvas: HTMLCanvasElement | null) => {
  const [position, setPosition] = useState({ x: 0, y: 0 });
  useEffect(() => {
    if (!canvas) return;
    const updatePosition = () => setPosition(getCanvasPositionRelativeToTheWholeScreen(canvas));
    updatePosition();
    window.addEventListener("resize", updatePosition);
    window.document.addEventListener("scroll", updatePosition);
    window.addEventListener("touchmove", updatePosition);
    window.addEventListener("touchend", updatePosition);
    // and also poll it every 100 ms
    const interval = setInterval(updatePosition, 100);
    return () => {
      window.removeEventListener("resize", updatePosition);
      window.document.removeEventListener("scroll", updatePosition);
      window.removeEventListener("touchmove", updatePosition);
      window.removeEventListener("touchend", updatePosition);
      clearInterval(interval);
    };
  }, [canvas]);
  return useMemo(() => position, [position.x, position.y]);
}

[bilowik]

In case someone else comes upon this, if you are utilizing web-sys in your bevy app, you can calculate the offset using something along the lines of:

#[derive(Resource, Default)]
pub struct TouchOffset(pub Vec2);

#[cfg(target_arch = "wasm32")]
pub fn update_touch_offset(
    mut touch_offset: ResMut<TouchOffset>,
) {
    if let Some(curr_offset) = web_sys::window()
        .and_then(|window| window.document())
        .and_then(|document| document.get_element_by_id("main-canvas")) // Assumes you set the id of the canvas to this
        .map(|element| element.get_bounding_client_rect())
        .map(|rect| Vec2::new(rect.left() as f32, rect.top() as f32)) {
        if curr_offset != touch_offset.0 {
            touch_offset.0 = curr_offset;
            info!("New touch offset calculated: {:?}", curr_offset);
        }
    }
    else {
        error!("Failed to get the canvas offset, touch inputs may be impacted");
    }
}

You would then subtract the TouchOffset from any touch coordinates you utilize. This probably doesn't need to run every frame though. Also the conditional is also not really needed, but I wanted to log when it changes while I was testing, you could just as easily set it without checking it.

(Also be aware that that error will spam your console if the element-getting logic is not right, so make sure you assign an id to the canvas and ensure bevy is rendering to that specific canvas.) (Edit: I ended up creating a Local counter to stop reporting the error after some amount of reports within a given timeframe, so that's an option for that for completeness)

Edit: I wrote up a plugin to do this offsetting automatically for TouchEvent and Touches resource that can be utilized until a proper fix finds its way in if anyone wants/needs a quick drop-in fix. Since it modifies the events themselves, this also fixes the issue for touch inputs and bevy UI elements, which my above example cannot. https://github.com/bilowik/bevy_wasm_touch_fix, also published on crates.io under the same name.

[daxpedda]

This should have been fixed by #10702. Let me know if you still encounter this issue.