Jetty-12 flakey HTTP2 test

[gregw]

Jetty version(s)

Flakey HTTP2 spec test in jetty-12

Description

Building locally on the command line, this is a frequent error.

[ERROR] Failed to execute goal org.mortbay.jetty:h2spec-maven-plugin:1.0.9:h2spec (h2spec) on project jetty-http2-tests: 
[ERROR] Failed test cases:
[ERROR]         [8.1.2.6 - Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length] failed
[ERROR]  Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
[ERROR]  Actual: RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
[ERROR] 

[lorban]

The latest version of h2spec's 8.1.2.6 test accepts RST_STREAM as a valid response: https://github.com/summerwind/h2spec/blob/master/http2/8_1_2_6_malformed_requests_and_responses.go#L69-L97 but according to the above error message, the test fails because GOAWAY was expected but RST_STREAM was received.

It seems we're using an out of date h2spec? @olamy could you please check that we're using the latest h2spec binary?

[lorban]

Running the latest h2spec v2.6.0 binary, 8.1.2.6 sometimes fails because it responds to the invalid request with a DATA frame instead of a RST_STREAM one, see the attached 8_1_2_6_failure.pcap.gz file for a capture of the failing run.

./h2spec -p 9999 http2/8.1.2.6
Hypertext Transfer Protocol Version 2 (HTTP/2)
  8. HTTP Message Exchanges
    8.1. HTTP Request/Response Exchange
      8.1.2. HTTP Header Fields
        8.1.2.6. Malformed Requests and Responses
          ✔ 1: Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length
          using source address 127.0.0.1:44348"content-length" header field which does not equal the sum of the multiple DATA frames payload length
          × 2: Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length
            -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR.
               Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
                         RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
                         Connection closed
                 Actual: DATA Frame (length:428, flags:0x01, stream_id:1)

Failures: 

Hypertext Transfer Protocol Version 2 (HTTP/2)
  8. HTTP Message Exchanges
    8.1. HTTP Request/Response Exchange
      8.1.2. HTTP Header Fields
        8.1.2.6. Malformed Requests and Responses
          using source address 127.0.0.1:44348
          × 2: Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length
            -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR.
               Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
                         RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
                         Connection closed
                 Actual: DATA Frame (length:428, flags:0x01, stream_id:1)

Finished in 2.0012 seconds
2 tests, 1 passed, 0 skipped, 1 failed
failed

so it looks like we do have a race condition somewhere.

[olamy]

The maven plugin is using the last available docker which is 2.6.0 see https://github.com/jetty-project/h2spec-maven-plugin/blob/5ac3283e1c426502610d7458a5adbf57dee1f643/src/main/java/org/mortbay/jetty/maven/h2spec/Http2SpecMojo.java#L136

[lorban]

Saw this failure in jenkins:

8.1.2.6. Malformed Requests and Responses
    using source address 172.17.0.8:40564
    × 1: Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length
    -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR.
        Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
                    RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
                    Connection closed
            Actual: DATA Frame (length:448, flags:0x01, stream_id:1)

I do not know how the reported 8.1.2.6 failure happened, but I'm considering it bogus and will assume this updated one is the actual one which should be worked on.

[gregw]

Can we work out the magic needed to exclude these tests for now, so we can get clean builds (but leave issue open until actually fixed)

[joakime]

The configuration has an excludeSpec configuration.

See: https://github.com/jetty-project/h2spec-maven-plugin/blob/5ac3283e1c426502610d7458a5adbf57dee1f643/src/main/java/org/mortbay/jetty/maven/h2spec/Http2SpecMojo.java#L98-L102

You might be able to use it from the command line with -Dh2spec.excludeSpecs=8.1.2.6 (the conversion from command line to List<String> in the config i'm not sure how to do that.

Meanwhile, I submitted PR #8228 to add that exclusion to the h2spec configuration.

[sbordet]

Fixed by #8228.