wpt icon indicating copy to clipboard operation
wpt copied to clipboard

Improve error message on wpt.live for accessibility tests which use testdriver.js

Open spectranaut opened this issue 1 year ago • 3 comments

The error message for wpt accessibility tests that use testdriver on wpt.live is kind of inscrutable (https://wpt.live/wai-aria/role/button-roles.html):

promise_test: Unhandled rejection with value: object "TypeError: window.test_driver_internal.get_computed_role is not a function"

I'm not sure if this is the best message, but I see that other testdriver.js functions, when not implemented, though a useful error message in this way.

Also I wonder if we could make clear in the documents somehow that not all tests can be run in wpt.live? I'm not sure where that information belongs, or I'd make the change. I thought about the home page of the documentation that links to wpt.live.. but I'm not sure the list of all "test types" that can't be run with wpt.live / ./wpt serve -- is it just the testdriver.js tests?

spectranaut avatar Apr 29 '24 23:04 spectranaut

ping @cookiecrook, what do you think of this change? If you think it's good (and weigh in on the error message) I'll make it as not-draft

spectranaut avatar Apr 29 '24 23:04 spectranaut

I don’t have a strong opinion but it seems fine to me. I’d defer this question to @gsnedders, @AutomatedTester, or @zcorpan.

cookiecrook avatar May 02 '24 06:05 cookiecrook

Also I wonder if we could make clear in the documents somehow that not all tests can be run in wpt.live? I'm not sure where that information belongs, or I'd make the change. I thought about the home page of the documentation that links to wpt.live.. but I'm not sure the list of all "test types" that can't be run with wpt.live / ./wpt serve -- is it just the testdriver.js tests?

Reftests are also not runnable, except by visually comparing with the reference in a separate tab. Tests that need HTTPS are also not so easy to run locally with wpt serve.

https://web-platform-tests.org/running-tests/from-web.html already has a statement about the limitations, but maybe the the home page could mention it as well.

zcorpan avatar May 06 '24 20:05 zcorpan