safari does not respect `421` status for HTTP/2 connection reuse / coalescing

[desimone]

What happened?

When using wildcard certificates, HTTP/2, Pomerium will respond with status 421 for different sites being proxied. Because of the way request coalescing works with HTTP/2, when we change the domain (authority) of the site we are proxying, (e.g. authenticate.x.y.z / verify.x.y.z / httpbin.x.y.z) Pomerium needs to tell the browser to reset the connection. However, it appears that Safari is not respecting that 421 and fails silently.

What did you expect to happen?

I expect to be able to use HTTP/2 and wildcard certificates across multiple domains for all major evergreen browsers.

Note, this only seems to effect safari.

How'd it happen?

1. Start with wildcard certs with no fronting proxy
2. Use safari 14.1
3. Go to verify.x.y.z
4. The callback to authenticate.x.y.z will fail as safari tries re-use the connection
5. Pomerium / envoy responds with a 421
6. Safari returns a blank empty response

What's your environment like?

• Safari Version 14.1 (16611.1.21.161.3)
• HTTP/2 enabled
• Wildcard certificates need to be used.
• Pomerium version (retrieve with pomerium --version): g91c7dc7
• Server Operating System/Architecture/Cloud: N/A (all forms)

What's your config.yaml?

N/A

What did you see in the logs?

{
  "level": "info",
  "service": "envoy",
  "upstream-cluster": "",
  "method": "GET",
  "authority": "authenticate.localhost.pomerium.io",
  "path": "/oauth2/callback",
  "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36",
  "referer": "https://accounts.google.com/",
  "forwarded-for": "192.168.1.63",
  "request-id": "4cd598e4-929f-4540-ab9f-c1359e6ac174",
  "duration": 0.471583,
  "size": 0,
  "response-code": 421,
  "response-code-details": "route_not_found",
  "time": "2021-04-28T12:17:14-07:00",
  "message": "http-request"
}

Additional context

• https://github.com/pomerium/pomerium/pull/1739
• https://tools.ietf.org/html/rfc7540#section-9.1.1
• https://tools.ietf.org/html/rfc7540#section-9.1.2 - section describing misdirected response
• https://bugs.chromium.org/p/chromium/issues/detail?id=954160#c5 - The bug was originally filed against Chromium however they indicated Istio was the issue.
• istio/istio#13589 - The bug was then filed against istio who indicated envoy was the issue.
• https://github.com/projectcontour/contour/issues/1493

Some options

• Support turning off HTTP/2
• ???

[calebdoxsey]

Unfortunately there's not really any good solution to this problem. Ways to fix it:

1. Add all the routes to every domain
2. Use different TLS certificates for each shared IP address
3. Use different IP addresses for each shared TLS certificate
4. Don't use safari
5. Don't use HTTP/2

[travisgroth]

Sounds like:

• add a global option to disable http2
• call out this type of problem in the docs for the http2 option
• disable http2 by default in all-in-one

[calebdoxsey]

Moving to the icebox until we have a better solution.