Connection fails when the Istio sidecar is deployed with an HTTPS/2 service

[ecefjga]

Is this a BUG or FEATURE REQUEST?: BUG

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: Y

Bug: Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version 0.7.1
kubectl version 1.10.0
kubernetes version 1.9.5

Is Istio Auth enabled or not ? Did you install the stable istio.yaml, istio-auth.yaml.... or if using the Helm chart please provide full command line input. stable istio.yaml without auth enabled.

What happened:

We are testing an scenario from POD (A) to POD (B), where the POD (A) is deployed without sidecar and it will send http2 secure requests to POD (B) which contains a HTTP2 server (based on Apache2 h2 secure server) and it has been deployed with a sidecar. According to information "when the Istio sidecar is deployed with an HTTPS service, the proxy automatically downgrades from L7 to L4 (no matter mTLS is enabled or not), which means it does not terminate the original HTTPS traffic. And this is the reason Istio can work on HTTPS services." However it's not working, from POD (A):

curl --http2 https://httpserver-istio:8012 --insecure -v
* Rebuilt URL to: https://httpserver-istio:8012/
*   Trying 10.96.112.186...
* TCP_NODELAY set
* Connected to httpserver-istio (10.96.112.186) port 8012 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* stopped the pause stream!
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

But if I deployed an HTTPS/1.1 server service with a sidecar in POD (B), it works fine sending HTTPS/1.1 requests from POD (A), the downgrade from L7 to L4 works in the proxy. Of course if I deployed the HTTPS/2 server service in POD (B) without sidecar, it works fine:

curl --http2 https://httpserver-k8s:8012 --insecure -v
* Rebuilt URL to: https://httpserver-k8s:8012/
*   Trying 10.98.227.40...
* TCP_NODELAY set
* Connected to httpserver-k8s (10.98.227.40) port 8012 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=ubuntu-mate
*  start date: Dec  4 15:51:32 2017 GMT
*  expire date: Dec  2 15:51:32 2027 GMT
*  issuer: CN=ubuntu-mate
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x20aaf80)
> GET / HTTP/2
> Host: httpserver-k8s:8012
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< date: Thu, 03 May 2018 19:45:41 GMT
< server: Apache/2.4.29 (Unix) LibreSSL/2.5.5
< last-modified: Wed, 14 Feb 2018 08:49:47 GMT
< etag: "11a-56528317e20c0"
< accept-ranges: bytes
< content-length: 282
< content-type: text/html
< 

What you expected to happen:

I expect the proxy automatically downgrades from L7 to L4 (no matter mTLS is enabled or not), which means it does not terminate the original HTTPS traffic and forward it to the HTTPS service for HTTP2 in the same way that currently does for HTTPS/1.1.

How to reproduce it:

We are using an Apache2 with h2 enabled for POD (B) HTTPS/2 service. But we have also tested with a simple go http2 server and we have got the same behavior.

Feature Request: N

Describe the feature:

[louiscryan]

Hmm. It looks like when the sidecar is enabled its interfering with the ALPN handshake.

@PiotrSikora I suspect this falls into the sniffing work you've been looking at.

[PiotrSikora]

@ecefjga could you share your configs for HTTP/1.1 and HTTP/2 deployments and services (or steps to reproduce)? Sidecar acting as L4 proxy doesn't differentiate HTTP/1.1 and HTTP/2 traffic, so I suspect there is something in the configs that differs between the two.

@louiscryan sniffing wasn't available in 0.7.1, so that's unrelated.