Support for Canvas/WebGL Interaction Beyond DOM Elements

[epeles]

Is your feature request related to a problem? Please describe.

I'm trying to use Chrome Devtools MCP to test a Three.js-based 3D application, but I've encountered a limitation where the MCP cannot interact with the canvas content itself - only with DOM elements.

Current Behavior

• The <canvas> element itself is in the DOM and can be found
• However, the actual 3D content rendered on the canvas (3D models, objects, interactive elements) cannot be interacted with
• Neither JavaScript evaluation tools nor mouse position tools seem to work for clicking specific coordinates on the canvas

Describe the solution you'd like

It would be helpful to have a way to:

1. Click at specific X/Y coordinates on the canvas (not just DOM elements)
2. Interact with the canvas content through pixel coordinates
3. Potentially use page.mouse.click(x, y) or similar low-level mouse interactions

Describe alternatives you've considered

Is this a known limitation of the MCP architecture, or is there a workaround I'm missing? Can the MCP:

• Use coordinate-based clicking (e.g., browser_mouse_click with pixel coordinates)?
• Execute Chrome Devtools low-level mouse API (e.g., page.mouse.click(x, y))?
• Interact with canvas elements beyond just finding them in the DOM?

Additional context

No response

[OrKoN]

Coordinate based clicking currently is known to not perform so well with the models but I think we can support it as an option.

[natorion]

Yeah, you would need to have a specialized model like computer_use to call our MCP server to make it work reliably.

Generally about interacting with canvas apps, the question is if you could not add a11y information to your three.js game e.g. with https://github.com/pmndrs/react-three-a11y