Tests for HTTP/1.1 Parsing (NUL, CR, LF, Colon and more)

[JannisBush]

Parsing (invalid) HTTP/1.1 responses is not fully defined in HTTP and Fetch and browser behavior diverges from the specs and between different browsers and there are not many tests in WPT yet. The aim of this pull request is to document the current browser behavior for various edge cases and start standardizing edge case behavior.

This pull requests adds a bunch of tests for (invalid) HTTP/1.1 responses for various edge cases. The tests are marked as tentative. The tests are about the general parsing of (invalid) HTTP/1.1 messages and are not about the parsing of individual (invalid) headers such as how browsers should handle invalid bytes in CSP headers (Pull Request for that here).

All tests use the following base response and inject various bytes in different places to test both status-line and header-line parsing. There are two types of tests:

• fetch tests: they fetch the resource and either fail (Network Error) or fetch the resource (Valid)
• framing tests: they frame the resource in an IFrame. They can result in Network Error (browser renders an error frame), active-XFO (browser renders an error frame due to XFO), or render the iframe (no-XFO, either the value is invalid or the full header line containing XFO is ignored). Note that the tests currently do not distinguish between Network Error and active-XFO.

HTTP/1.1 200 OK
X-Frame-Options: DENY
Content-Length: 5

Test.

The following shows all the injection points and their names:

<first>HT<in-http>TP<before-slash>/<after-slash>1<before-dot>.<after-dot>1<after-ver> <before-num>200<after-num> <before-reason>OK<after-reason>
<leading>X-Frame<in-name>-Options<before-colon>:<after-colon>DE<in-value>NY<after-value>
Content-Length: 5

Test.

The following bytes are injected: LF, CR, HTAB, SP, NUL, COLON.

The expected behavior is described in a mapping:

let expected = {
    "SP": {
        "first": ["Network Error", "Network Error"],
        ...
        "in-value": ["Valid", "no-XFO"],
        "after-value": ["Valid", "active-XFO"]
    },
    ...
}

Currently, there are 102*2=204 tests. In the future, other characters could be injected or other tests can be added. The initial expected mapping is strict and expects "Network Error" for almost everything that is not explicitly allowed. The current results are (both on HTTP and HTTPS):

• Chrome: 114 Pass, 90 Fail
• Firefox: 61 Pass, 143 Fail
• Safari: 92 Pass, 112 Fail

---

See also:

• Fetch overview issue of HTTP/1.1 parsing
• Define behavior for invalid header names
• Related bug reports in Safari: https://bugs.webkit.org/show_bug.cgi?id=285661, https://bugs.webkit.org/show_bug.cgi?id=285606, https://bugs.webkit.org/show_bug.cgi?id=272749, https://bugs.webkit.org/show_bug.cgi?id=272748, https://bugs.webkit.org/show_bug.cgi?id=272740
• Related bug reports in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1891719

[annevk]

Thank you for writing these, could you please address the lint issue?

One way of addressing it might be to instead of having lots of .asis files generate those dynamically on the server as some other HTTP parsing tests are doing. But you might also be able to exclude a whole directory for a specific rule, maybe?

[JannisBush]

Thank you for writing these, could you please address the lint issue?

One way of addressing it might be to instead of having lots of .asis files generate those dynamically on the server as some other HTTP parsing tests are doing. But you might also be able to exclude a whole directory for a specific rule, maybe?

I fixed the linting errors (mostly by adding them to the correct ignore locations). I intentionally generated the asis files such that one can easily look at the exact bytes send with a hex editor.

[jdm]

The end of the log at https://community-tc.services.mozilla.com/tasks/WDE8qX2hSEi6Srt9TIiQuQ/runs/0/logs/public/logs/live.log points out various test results that appear to be unstable.

[JannisBush]

The end of the log at https://community-tc.services.mozilla.com/tasks/WDE8qX2hSEi6Srt9TIiQuQ/runs/0/logs/public/logs/live.log points out various test results that appear to be unstable.

The issue is that there is no reliably way to check whether a frame finished loading/resulted in an error and the test relies on a timeout. I copied the approach from here: https://github.com/web-platform-tests/wpt/blob/29d1210ae907e5bace1ecf047efde527c6377069/fetch/h1-parsing/resources-with-0x00-in-header.window.js#L16C1-L22C12

I increased the timeout from 1s to 2s and hope that it now results in stable results

[jdm]

Mmm, still quite a bit of instability in https://community-tc.services.mozilla.com/tasks/DoF4XaEySBSPqEBcjKMm8g/runs/0/logs/public/logs/live.log :(

[JannisBush]

Mmm, still quite a bit of instability in https://community-tc.services.mozilla.com/tasks/DoF4XaEySBSPqEBcjKMm8g/runs/0/logs/public/logs/live.log :(

Right. Probably because we are running a lot of framing tests on the same top-level page. Should I simply increase the timeout to a higher value (e.g., 5s) or should we try to split the tests and run them in batches?

[annevk]

I intentionally generated the asis files such that one can easily look at the exact bytes send with a hex editor.

I think it would be easier to analyze failures if one didn't have to do that and you could just see them written out in JSON as escapes. Converted to actual bytes at the last moment.

It probably makes sense to split this file using variants, yeah.

[JannisBush]

I intentionally generated the asis files such that one can easily look at the exact bytes send with a hex editor.

I think it would be easier to analyze failures if one didn't have to do that and you could just see them written out in JSON as escapes. Converted to actual bytes at the last moment.

I had some issues with the other handlers and only the asis handler enabled me to send exactly the responses I wanted to generate. The name of the file also includes the description of what is included where and one only needs to look at the file if one wants to verify that the files are correct.

It probably makes sense to split this file using variants, yeah.

Done

Seems like even with variants (max 6 frames per test page) and higher timeouts (4s), the results are still unstable in Chrome (stable in Firefox). I tried to reproduce it locally but failed. There might be another issue here or Chrome itself is unstable here.

[annevk]

@foolip @past are there known stability issues with Chromium?

[past]

@foolip @past are there known stability issues with Chromium?

All the existing tests seem pretty solid on Chromium, if you click on the test name and then click on Show History Timeline. These failures are all specific to the new test, but I don't know why that is the case.

[JannisBush]

Could it be that there is an issue with the test run infrastructure?

I reduced the timeout from 4s to 100ms and ran 30 stability runs several times and the results are always stable locally: python3 ./wpt run --channel=dev --verify --verify-no-chaos-mode --verify-repeat-loop=0 --verify-repeat-restart=30 --log-mach-level=info --log-mach=- -y --no-pause --no-restart-on-unexpected --install-fonts --verify-log-full --no-headless --enable-swiftshader --install-browser --install-webdriver chrome fetch/h1-parsing

[annevk]

One thing you could try is to rebase on master and force push. That would at least guarantee that everything is the latest.