Invalid authority header from envoy grpc client

[placydo]

Hey,

We have been using image with tag next with great success. As we noticed You have released newer version v0.12.2 we have tried to use it and hit some wall. We are using projectcontour as ingress controller. The issue does persist with old and new version of it. Currently we are using: contour:v1.20.1 envoy:v1.21.1 At envoy I can see following logs: {"path":"/topsecret","protocol":"HTTP/2","upstream_host":null,"user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36","upstream_service_time":null,"requested_server_name":"zz.com","response_flags":"UAEX","upstream_cluster":"topsecret_expose-es-console_80","downstream_remote_address":"xx.xx.xxx.x:63745","request_id":"966e13ae-844a-4c8e-9a8e-c686dd7eaed6","method":"GET","@timestamp":"2022-04-11T18:47:02.238Z","upstream_local_address":null,"authority":"zz.com","bytes_sent":0,"bytes_received":0,"duration":0,"response_code":403,"uber_trace_id":null,"x_forwarded_for":"xx.xx.xxx.x","downstream_local_address":"xx.xx.xxx.x:8443"} Also noticed that You did add additional debug flag (without it there are no new logs reported by EAS) so we have added it and following logs do show up:

Http2Session server (117) i/o stream consumed
Http2Session server (117) scheduling write
Http2Session server (117) sending pending data
Http2Session server (117) nghttp2 has 9 bytes to send
Http2Session server (117) wants read? 1
Http2Session server (117) receiving 1845 bytes, offset 0
Http2Session server (117) receiving 1845 bytes [wants data? 1]
Http2Session server (117) complete frame received: type: 4
Http2Session server (117) complete frame received: type: 8
Http2Session server (117) beginning headers for stream 1
Http2Session server (117) Error 'Invalid HTTP header field was received: frame type: 1, stream: 1, name: [:authority], value: [extension/ingress/eas-external-auth-server]'
Http2Session server (117) invalid frame received (0/1000), code: -531
Http2Session server (117) sending pending data
Http2Session server (117) nghttp2 has 9 bytes to send
Http2Session server (117) stream 1 closed with code: 1
HttpStream 1 (121) [Http2Session server (117)] closed with code 1
HttpStream 1 (121) [Http2Session server (117)] destroying stream
Http2Session server (117) nghttp2 has 13 bytes to send
Http2Session server (117) wants read? 1
Http2Session server (117) wants read? 1
HttpStream 1 (121) [Http2Session server (117)] tearing down stream
Http2Session server (117) receiving 9 bytes, offset 0
Http2Session server (117) receiving 9 bytes [wants data? 1]
Http2Session server (117) complete frame received: type: 4
Http2Session server (117) settings refreshed for session
Http2Session server (117) sending pending data
Http2Session server (117) wants read? 1

Httpproxy kubernetes object:

kind: HTTPProxy
metadata:
  name: zz
  namespace: contour
spec:
  includes:
  - conditions:
    - prefix: /topsecret
    name: somename
    namespace: somenamespace
  virtualhost:
    authorization:
      authPolicy:
        context:
          x-eas-verify-params: '{"config_token":"REDACTED"}'
          x-forwarded-port: "443"
      extensionRef:
        name: eas-external-auth-server
        namespace: contour
      responseTimeout: 5s
    fqdn: zz.com
    tls:
      minimumProtocolVersion: "1.3"
      secretName: secret-tls

Is there any way we could make the newest EAS to work with projectcontour? Thanks

[travisghansen]

Until this is fixed upstream you may use this deployed as a sidecar (built to overcome other projects with the exact same issue): https://github.com/democratic-csi/csi-grpc-proxy

EDIT: scratch that, I forgot this was implemented specifically to handle h2c (generally used with uds) so it probably won't work for ssl connections unless you put yet another proxy like stunnel in front

[travisghansen]

Linking to contour issue:

https://github.com/projectcontour/contour/issues/4278

[travisghansen]

Did this get fixed upstream?

[sunjayBhatia]

this change in Contour should have fixed the issue: https://github.com/projectcontour/contour/commit/3fc56275f51db51c64d33f50e052efd4de3349de

released in v1.22.0

[travisghansen]

Great! @placydo can you confirm that it works end-to-end with all the fixes?

[placydo]

@sunjayBhatia @travisghansen It does look like it is fixed. Thanks You Travis for all heart put into the project and and hard work to try to resolve this issue.