wpt
wpt copied to clipboard
web-platform-tests
Add ability to run servers on different IP addresses.
This PR adds two new alternate_hosts to the default test configuration:
private:private-web-platform.testpublic:public-web-platform.test
These are hosted by a collection of new servers bound to different IP addresses (127.1.0.1 and 127.2.0.1 respectively). All protocols and subdomains are supported. This results in a ~3x increase in the size of ./wpt make-hosts-file's output.
These new IP addresses are configurable using a new config property: alternate_ip_addresses, which maps host names to IP addresses.
Sending this for review now, but there is one last thing I need to fix: this probably does not work macOS, as noted in https://github.com/web-platform-tests/rfcs/issues/79.
Seems that macOS indeed failed to start the additional servers. Windows on the other hand seems to have failed some tests due to them overriding alternate_hosts with a value that does not contain private and public keys.
Attempted to fix Windows test failures.
Along the way, I changed the way the server config is passed to WebDriver tests. Instead of serializing the whole config in an environment variable, I amended the code to dump the config as JSON in a temporary file (the tests are conveniently executed against a temporary directory) and pass the file path in the environment variable instead. This avoids repeatingly bumping into the 16-bit environment variable length limitation on Windows, and the need to special-case origin-policy subdomains.
I also added the py39 environment to tools/tox.ini so that I could run the tests on my machine, which does not have python 3.6-3.8 :)
So my high level impression is that this looks quite reasonable. It might make sense to break the WebDriver config change into a seperate PR since that's somewhat orthogonal (and much less risky) than the rest of the changes. I think we're going to need to run this one through a full trigger run and in vendor CI to ensure we're not accidentially breaking anything.
@letitz do you know how you'd run this through Chromium's CI to look for regressions? If not let me know and I'm happy to chat. (We might have to update our copy of tools/ for the patch to apply and for the comparison to be meaningful.)
@foolip I'm not sure.
My first thought was that I could download a patch version of this PR, apply it in a Chomium client and run CI jobs on the resulting patch. However, the tools/ folder does not exist in Chromium's mirror of the wpt repo: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/. I'm not sure how or even if Chromium relies on wptserve and wptrunner.
How do you maintainers usually do this?
Given https://github.com/web-platform-tests/rfcs/pull/83 and https://github.com/web-platform-tests/wpt/pull/28870, is this now obsolete and can be closed?
