results-collection
results-collection copied to clipboard
web-platform-tests
Support mobile browsers
This came up at the web-platform-tests breakout session at TPAC: https://www.w3.org/2017/11/08-testing-minutes.html
I think that as a baseline, we'd want Chrome on Android, Safari on iOS and Firefox on Android. @thejohnjansen, how about Edge?
breezet is here:)
It seems really hard to let every browser implement WebDriver API. So can we find other way to sovle this problem. For examples can we build a common driver to proxy a browser witch not support WebDriver API.
wptrunner already provides an abstraction layer here. Each executor has to implement the interface at https://github.com/w3c/web-platform-tests/blob/master/tools/wptrunner/wptrunner/executors/base.py#L100. That more or less just requires a run_test method that loads the test and gets results out. For example in the case of servo it's implemented by literally starting the browser with the test url and reading test results from stdout.
Of course if you want to implement testdriver then a more comprehensive browser control API is required, but that's a tiny fraction of all tests at this point.
here is a implementation: https://github.com/web-driverify/web-driverify
But if it want to run in testdriver, maybe we have some works need to do for this.
@tayqassqan is it primarily the Baidu browser on Android that we're talking about here? What options to web developers have for automatically starting the browser, doing something, and getting a result back? I think that if there's any mechanism for this, then we could get this off the ground.
If you did this, how often do you think you'd be able to do full runs? If 10 full runs a day would be a stretch, then https://github.com/w3c/wptdashboard/issues/164 would be a limit, and we'd have to work out a way of dealing with some set of browsers getting results less frequently. Of course we'd also have to work out which browsers to show by default if the total number of configurations were to increase as much as we're hoping. This is all stuff we can work out.
@tayqassqan is it primarily the Baidu browser on Android that we're talking about here?
I think it is not a goal for this issue. I expect things is most mobile browsers in China can join WPT.
What options to web developers have for automatically starting the browser, doing something, and getting a result back? I think that if there's any mechanism for this, then we could get this off the ground.
I'm not sure is it a good mechanism on WPT now, we will use web-driverify to do more testings on our own work, and accumulate more experience for this.
If 10 full runs a day would be a stretch, then #164 would be a limit, and we'd have to work out a way of dealing with some set of browsers getting results less frequently.
That may be a question that has to be considered in order to show the WPT results faster. If we verify the mechanism, we can look back at the problem.
Finally, thank you for your suggestion, and hopefully keep communicating.
@mattl having Edge Android would be cool, but if they're telling the truth about using unmodified blink then we shouldn't learn anything (should match Chrome Android exactly for the corresponding version). Other chromium browsers (UC, Samsung, even maybe slightly Opera) would be more likely to have some difference from Chrome.
As for iOS, it's even harder for the browser to have web-exposed behavior that's different from Safari (store policy requires browsers to use the built in WebKit). We know Chrome iOS does inject some JavaScript which can result in behavior differences, so I'd like us to get that going (@alijuma is looking into this). I'm not sure which others would be interesting.
I think it is not a goal for this issue. I expect things is most mobile browsers in China can join WPT.
@tayqassqan, I'd very much like that, and to have every browser with significant number of users represented.
Right now the setup is that we (Google/Bocoup) do all the runs. That will not scale to a very large number of browsers, and I don't want our willingness to run to be required in order to get results in. So, I think we need to figure out how to have "external" runs, where the results are posted back somehow. And to get that working we need someone to cooperate with on the first such browser. If the Baidu team is eager to do that, that'd be great, but maybe let's chat offline first.
@jensl FYI, this is very much the same work that would be required to get Opera (desktop or mobile) results into the dashboard.
If the Baidu team is eager to do that, that'd be great, but maybe let's chat offline first.
@foolip We are very willing to make this attempt.
I'll send you an email in the last two days, let's communicate in detail on email.
I'll send you an email in the last two days, let's communicate in detail on email.
Didn't see anything yet :)