full-stack-fastapi-template icon indicating copy to clipboard operation
full-stack-fastapi-template copied to clipboard

Published 20 hours ago • verified publisher icon fastapi

Force HTTP1 or enable HTTP2

Open abrichr opened this issue 3 years ago • 5 comments

We have deployed according to the steps in https://github.com/tiangolo/full-stack-fastapi-postgresql/issues/322.

However, we have a problem that only seems to affect some clients:

$ curl -v 'https://stag.foo.com/'
*  Trying <ip>...
* TCP_NODELAY set
* Connected to stag.foo.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
 CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=stag.foo.com
* start date: Feb 23 01:50:57 2021 GMT
* expire date: May 24 01:50:57 2021 GMT
* subjectAltName: host "stag.foo.com" matched cert's "stag.foo.com"
* issuer: <bar>
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd84900d600)
> GET / HTTP/2
> Host: stag.foo.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host stag.foo.com left intact
curl: (16) Error in the HTTP2 framing layer
* Closing connection 0

Notice the error:

* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.

Most clients do not experience this, however:

% curl -v 'https://stag.foo.com/'
*   Trying <ip>...
* TCP_NODELAY set
* Connected to stag.foo.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=stag.foo.com
*  start date: Mar 15 02:07:24 2021 GMT
*  expire date: Mar 29 02:07:24 2021 GMT
*  subjectAltName: host "stag.foo.com" matched cert's "stag.foo.com"
*  issuer: <bar>
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: stag.foo.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 3618
< Content-Type: text/html
< Date: Mon, 15 Mar 2021 17:48:39 GMT
< Etag: "604f7271-e22"
< Last-Modified: Mon, 15 Mar 2021 14:42:57 GMT
< Server: nginx/1.15.12
< 
* Connection #0 to host stag.foo.com left intact
<!DOCTYPE html><html lang=en>...</html>* Closing connection 0

The difference from the failing output appears to start here:

> GET / HTTP/1.1

Therefore, when the request succeeds, it's because the client falls back to HTTP/1.1. When the request fails, it's because the client attempts to use HTTP/2, which the server claims to support but fails to do so.

The solution appears to be either:

  1. Force HTTP/1.1, or
  2. Enable HTTP/2

How can we configure this stack to force HTTP/1.1 or enable HTTP/2?

abrichr avatar Mar 16 '21 22:03 abrichr

This appears to be a known issue:

https://github.com/traefik/traefik/issues/6392

https://github.com/traefik/traefik/issues/7776

https://github.com/traefik/traefik/issues/7814

One workaround (described in https://community.traefik.io/t/traefik-advertises-http-2-when-backend-doesnt-support-it/1776) appears to involve creating a traefik.toml file for "dynamic" configuration with the following:

[tls.options]
  [tls.options.foo]
    minVersion = "VersionTLS12"
    cipherSuites = [
      "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "TLS_RSA_WITH_AES_256_GCM_SHA384"
    ]

Here this would probably also require the addition of something like the following to traefik.yml and/or docker-compose.yml under the traefik image:

  ...
  command:
    - --providers.file
    - --providers.docker
    ...

Another workaround (described in https://community.traefik.io/t/traefikv2-http-2-0/1199) involves setting the following environment variables:

GODEBUG=http2client=0  # disable HTTP/2 client support
GODEBUG=http2server=0  # disable HTTP/2 server support

Here this would involve the addition of the following to traefik.yml and/or docker-compose.yml under the traefik image:

  ...
  environment: 
    - GODEBUG=http2client=0
    - GODEBUG=http2server=0
  ...

Will test these out and report back.

abrichr avatar Mar 19 '21 21:03 abrichr

Adding the variables in both docker-compose.yml and traefik.yml yielded the following change:

$ curl  -v 'https://stag.foo.com/'
*   Trying <ip>...
* TCP_NODELAY set
* Connected to stag.foo.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, close notify (256):
* LibreSSL SSL_connect: SSL_ERROR_ZERO_RETURN in connection to stag.foo.com:443 
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_ZERO_RETURN in connection to stag.foo.com:443

abrichr avatar Mar 19 '21 22:03 abrichr

Tried various combinations of specifying minVersion and cipherSuites in the .toml file and in the traefik.yml, with no luck.

Turning on debug logging in traefik.yml shows this whenever an attempt is made to connect from the affected machine:

[email protected]    | time="2021-03-19T23:16:37Z" level=debug msg="http: TLS handshake error from 10.0.0.2:48084: read tcp 10.0.0.28:443->10.0.0.2:48084: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:37Z" level=debug msg="http: TLS handshake error from 10.0.0.2:18313: read tcp 10.0.0.28:443->10.0.0.2:18313: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:37Z" level=debug msg="http: TLS handshake error from 10.0.0.2:51018: read tcp 10.0.0.28:443->10.0.0.2:51018: read: connection reset by peer"
[email protected]    | time="2021-03-19T23:16:37Z" level=debug msg="http: TLS handshake error from 10.0.0.2:50130: read tcp 10.0.0.28:443->10.0.0.2:50130: read: connection reset by peer"
[email protected]    | time="2021-03-19T23:16:37Z" level=debug msg="http: TLS handshake error from 10.0.0.2:22532: read tcp 10.0.0.28:443->10.0.0.2:22532: read: connection reset by peer"
[email protected]    | time="2021-03-19T23:16:37Z" level=debug msg="http: TLS handshake error from 10.0.0.2:24431: read tcp 10.0.0.28:443->10.0.0.2:24431: read: connection reset by peer"
[email protected]    | time="2021-03-19T23:16:37Z" level=debug msg="http: TLS handshake error from 10.0.0.2:33427: read tcp 10.0.0.28:443->10.0.0.2:33427: read: connection reset by peer"
[email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:50998: read tcp 10.0.0.28:443->10.0.0.2:50998: read: connection reset by peer"                                                               
[email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:23661: read tcp 10.0.0.28:443->10.0.0.2:23661: read: connection reset by peer"                                                                
[email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:29784: read tcp 10.0.0.28:443->10.0.0.2:29784: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:27384: read tcp 10.0.0.28:443->10.0.0.2:27384: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:18647: read tcp 10.0.0.28:443->10.0.0.2:18647: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:24732: read tcp 10.0.0.28:443->10.0.0.2:24732: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:38617: read tcp 10.0.0.28:443->10.0.0.2:38617: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:52858: read tcp 10.0.0.28:443->10.0.0.2:52858: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:22284: read tcp 10.0.0.28:443->10.0.0.2:22284: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:63197: read tcp 10.0.0.28:443->10.0.0.2:63197: read: connection reset by peer"                                                                                                                                                   [email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:19347: read tcp 10.0.0.28:443->10.0.0.2:19347: read: connection reset by peer"
[email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:53124: read tcp 10.0.0.28:443->10.0.0.2:53124: read: connection reset by peer"
[email protected]    | time="2021-03-19T23:16:38Z" level=debug msg="http: TLS handshake error from 10.0.0.2:44466: read tcp 10.0.0.28:443->10.0.0.2:44466: read: connection reset by peer"

Also:

# docker ps -q | xargs -n 1 docker inspect --format '{{ .Name }} {{range .NetworkSettings.Networks}} {{.IPAddress}}{{end}}' | sed 's#^/##';
traefik_traefik.1.r3zh0w7cfsg7w5zol9dx1qau0  10.0.0.28 10.0.1.252
stag-foo-com_backend.1.lrcsoc0ibiz3amy0yubxdcj0l  10.0.3.84
stag-foo-com_frontend.1.y1n5bhd3hcf8n77dxs3bhb7t9  10.0.3.85
stag-foo-com_proxy.1.xn37r30go8hzday4q3e3uvih5  10.0.3.83 10.0.1.250
stag-foo-com_celeryworker.1.j46cextpdnkfej9ab8cxrw9i2  10.0.3.82
stag-foo-com_pgadmin.1.9c0jl2tpyrx9wq829btm9u8uv  10.0.3.71 10.0.1.229
stag-foo-com_flower.1.op2nbhf1p4ib7bje1cc89avy0  10.0.3.70 10.0.1.228
stag-foo-com_db.1.izbb55hvj0s6fxg9798fplauf  10.0.3.69
stag-foo-com_queue.1.aft7z4mb56ivfzkgkwu2r4jdh  10.0.3.59

None of the container IPs match the target in the Traefik log of 10.0.0.2... 🤔

abrichr avatar Mar 19 '21 23:03 abrichr

Same for Traefik v2.9.6: traefik.tcp.routers.gf7tcp.tls.passthrough=true results in error "404 not found" in Microsoft Edge.

In Internet Explorer 11 (no HTTP/2 ?) TLS passthrough to Glassfish 7 (HTTP/2 enabled)

Log has no handshake error message - Traefik has the correct IP & port in dashboard.

Setting TLS options does not help, loglevel DEBUG does not print any error message.

But I only have this problem in a docker swarm environment. In docker standalone the same docker image (traefik & glassfish) works fine with

      - "traefik.tcp.routers.gf.entrypoints=https"
      - "traefik.tcp.routers.gf.rule=HostSNI(`test2.informatik.pd.intranet.bs.ch`)"
      - "traefik.tcp.routers.gf.tls.passthrough=true"
      - "traefik.tcp.routers.gf.service=gf-service"
      - "traefik.tcp.services.gf-service.loadbalancer.server.port=8181"

Very strange behaviour.

Edit: It might have something to do with HTTP/2 and wildcard certificates. I will change that and report.

pdbs avatar Dec 20 '22 19:12 pdbs

It seems that adding

[tls.options]
  [tls.options.default]
    alpnProtocols = ["http/1.1", "h2"]

to dynamic configuration file solves the problem, but only if the certificate is exclusively used on the tcp router (and not on other https routers) since it contains wildcard DNS SANs and this can not be handled by traefik. Maybe adding a configuration value to [[tls.certificates]] which indicates all hostnames the certificate should be used for could help - or defining the certificate file on service level value.

I assume the problem is not (only) http2 in traefik since browsers acted differently: Sometimes they where showing the webserver's index.html, sometimes 404 from traefik although different certificates were used. But setting the order "http/1.1", "h2" solved the problem. curl always worked, adding GODEBUG env variables worked for the tcp router but left the http routers unusable.

pdbs avatar Dec 22 '22 07:12 pdbs