Envoy disconnects because of TLSV1_ALERT_PROTOCOL_VERSION

[laugmanuel]

What steps did you take and what happened: After deploying this project using the provided resources and creating an extensionservice to target this authserver, the envoy sends the requests there but receives a TLSV1_ALERT_PROTOCOL_VERSION and rejects every connection with UAEX.

The request doesn't even get logged by contour-authserver.

❯ kubectl logs envoy-jxg5p -c envoy
[...]
[2021-12-07 15:11:07.906][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:101] starting async DNS resolution for 127.0.0.1
[2021-12-07 15:11:07.906][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:109] async DNS resolution complete for 127.0.0.1
[2021-12-07 15:11:07.906][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:155] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2021-12-07 15:11:07.975][1][debug][main] [source/server/server.cc:199] flushing stats
[2021-12-07 15:11:07.994][13][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:79] tls inspector: new connection accepted
[2021-12-07 15:11:08.007][13][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:150] tls:onServerName(), requestedServerName: example.app.k8s.domain
[2021-12-07 15:11:08.008][13][debug][conn_handler] [source/server/connection_handler_impl.cc:501] [C304] new connection
[2021-12-07 15:11:08.199][1][debug][upstream] [source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 100.75.36.192:8001
[2021-12-07 15:11:08.199][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for contour, refresh rate 5000 ms
[2021-12-07 15:11:08.314][13][debug][http2] [source/common/http/http2/codec_impl.cc:1184] [C304] updating connection-level initial window size to 268435456
[2021-12-07 15:11:08.314][13][debug][http] [source/common/http/conn_manager_impl.cc:254] [C304] new stream
[2021-12-07 15:11:08.315][13][debug][http] [source/common/http/conn_manager_impl.cc:886] [C304][S4958010076065905179] request headers complete (end_stream=true):
':method', 'GET'
':path', '/'
':scheme', 'https'
':authority', 'example.app.k8s.domain'
'authorization', 'Basic <basic-auth-header>'
'user-agent', 'curl/7.64.1'
'accept', '*/*'

[2021-12-07 15:11:08.315][13][debug][http] [source/common/http/filter_manager.cc:755] [C304][S4958010076065905179] request end stream
[2021-12-07 15:11:08.316][13][debug][lua] [source/extensions/filters/common/lua/lua.cc:39] coroutine finished
[2021-12-07 15:11:08.316][13][debug][router] [source/common/router/router.cc:425] [C0][S326086556289697609] cluster 'extension/ingress/contour-authserver-htpasswd' match for URL '/envoy.service.auth.v3.Authorization/Check'
[2021-12-07 15:11:08.316][13][debug][router] [source/common/router/router.cc:582] [C0][S326086556289697609] router decoding headers:
':method', 'POST'
':path', '/envoy.service.auth.v3.Authorization/Check'
':authority', 'extension/ingress/contour-authserver-htpasswd'
':scheme', 'https'
'te', 'trailers'
'grpc-timeout', '200m'
'content-type', 'application/grpc'
'x-envoy-internal', 'true'
'x-forwarded-for', '100.64.2.77'
'x-envoy-expected-rq-timeout-ms', '200'

[2021-12-07 15:11:08.317][13][debug][pool] [source/common/http/conn_pool_base.cc:79] queueing stream due to no available connections
[2021-12-07 15:11:08.317][13][debug][pool] [source/common/conn_pool/conn_pool_base.cc:106] creating a new connection
[2021-12-07 15:11:08.317][13][debug][client] [source/common/http/codec_client.cc:41] [C305] connecting
[2021-12-07 15:11:08.317][13][debug][connection] [source/common/network/connection_impl.cc:860] [C305] connecting to 100.64.2.76:9443
[2021-12-07 15:11:08.317][13][debug][connection] [source/common/network/connection_impl.cc:876] [C305] connection in progress
[2021-12-07 15:11:08.317][13][debug][http2] [source/common/http/http2/codec_impl.cc:1184] [C305] updating connection-level initial window size to 268435456
[2021-12-07 15:11:08.318][13][debug][connection] [source/common/network/connection_impl.cc:666] [C305] connected
[2021-12-07 15:11:08.319][13][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:224] [C305] TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2021-12-07 15:11:08.319][13][debug][connection] [source/common/network/connection_impl.cc:241] [C305] closing socket: 0
[2021-12-07 15:11:08.320][13][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:224] [C305] TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2021-12-07 15:11:08.320][13][debug][client] [source/common/http/codec_client.cc:99] [C305] disconnect. resetting 0 pending requests
[2021-12-07 15:11:08.320][13][debug][pool] [source/common/conn_pool/conn_pool_base.cc:343] [C305] client disconnected, failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2021-12-07 15:11:08.320][13][debug][router] [source/common/router/router.cc:1026] [C0][S326086556289697609] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
[2021-12-07 15:11:08.321][13][debug][http] [source/common/http/async_client_impl.cc:101] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436526:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION'

[2021-12-07 15:11:08.321][13][debug][http] [source/common/http/filter_manager.cc:839] [C304][S4958010076065905179] Sending local reply with details ext_authz_error
[2021-12-07 15:11:08.321][13][debug][http] [source/common/http/conn_manager_impl.cc:1484] [C304][S4958010076065905179] encoding headers via codec (end_stream=true):
':status', '403'
'date', 'Tue, 07 Dec 2021 15:11:07 GMT'
'server', 'envoy'

[2021-12-07 15:11:08.321][13][debug][http2] [source/common/http/http2/codec_impl.cc:964] [C304] stream closed: 0
[2021-12-07 15:11:08.478][13][debug][connection] [source/common/network/connection_impl.cc:634] [C304] remote close
[2021-12-07 15:11:08.478][13][debug][connection] [source/common/network/connection_impl.cc:241] [C304] closing socket: 0
[2021-12-07 15:11:08.478][13][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:308] [C304] SSL shutdown: rc=1
[2021-12-07 15:11:08.478][13][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:224] [C304]
[2021-12-07 15:11:08.479][13][debug][conn_handler] [source/server/connection_handler_impl.cc:164] [C304] adding to cleanup list

What did you expect to happen:

Valid response by contour-authserver.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Software version:
  • contour-authserver: master branch
  • contour: 1.12.0
  • envoy: 1.17.0
• Kubernetes version: (use kubectl version): 1.20.9
• Kubernetes installer & version: n/a
• Cloud provider or hardware configuration: n/a
• OS (e.g. from /etc/os-release): n/a

[stevesloka]

Is the auth-server healthy? Is does it have a valid cert applied? It looks like Envoy tried to reach out to the auth server and failed with that TLS error.

[laugmanuel]

@stevesloka, yes the authserver was healthy and using a self-signed cert (as per https://projectcontour.io/guides/external-authorization/).

[ianlewis]

I've run into this with envoy as well. I also get it when I try to connect via grpc_cli.

$ grpc_cli -channel_creds_type=ssl -ssl_client_key tls.key -ssl_client_cert combined.crt -proto_path . -protofiles envoy/service/auth/v3/external_auth.proto call localhost:9443 Check
E0310 06:51:13.252021137       1 ssl_transport_security.cc:1233] Handshake failed with fatal error SSL_ERROR_SSL: error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION.
E0310 06:51:13.287141830       1 ssl_transport_security.cc:1233] Handshake failed with fatal error SSL_ERROR_SSL: error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION.
connecting to localhost:9443
E0310 06:51:13.313377033       1 cli_call.cc:69]             assertion failed: ok

[youngnick]

This error is saying that the server doesn't want to talk the TLS version you are trying to use. Contour recently changed its TLS support to default to TLS version 1.2 for many things, we may need to update something in the authserver.

[joda01]

I have the same problem. It seems that the auth server expects TLS version v3 but envoy tries to connect with TLS version V1 https://github.com/projectcontour/contour-authserver/blob/947288352f4e8fe2357f22177ffe58e73546a801/pkg/auth/server.go#L120

[joda01]

@laugmanuel Using auth server v2 fixed the problem for me

[chudesno]

Thanks for a hint @joda01, v2 works fine!