Use Case - Front Proxy with mTLS

[jasonmccallister]

Last meeting we discussed providing some use cases on how people are deploying Envoy in production. We have somewhat of an interesting use case I thought I would share.

We deploy Envoy as a front proxy that sits behind a CDN. Our Envoy Proxy sits behind a network load balancer and has one TLS listener that requires all connections present a client certificate signed by the Cloudflare CA. Many CDN providers, such as Cloudflare and Fastly, provide a way to authenticate origin requests using a CA/self-signed certificate. See the Cloudflare Authenticated Origin Pulls documentation for additional details. This provides similar benefits as deploying Envoy as a sidecar with mTLS; only requests that originate from the CDN provider with a valid certificate will be accepted by the Envoy Proxy.

This affords a few benefits as a front proxy:

1. Requires all requests come from the CDN provider (Cloudflare, etc.). If someone does manage to discovery a load balancer DNS record they cannot spoof headers/etc.
2. Forcing requests through the CDN provider also ensures requests are filtered by the WAF and DDOS protection is applied by the CDN.

Happy to discuss any specific challenges on getting this deployed and configured correctly, but this is one of our use cases.

Example

An example listener would look something like this:

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 8443
      listener_filters:
        - name: "envoy.filters.listener.tls_inspector"
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
      per_connection_buffer_limit_bytes: 32768 # 32 KiB
      access_log:
        - name: envoy.access_loggers.stdout
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                codec_type: AUTO
                stat_prefix: ingress_tls
                rds:
                  route_config_name: ingress_tls
                  config_source:
                    api_config_source:
                      api_type: REST
                      transport_api_version: V3
                      cluster_names:
                        - xds_proxy
                      refresh_delay: 10s
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              require_client_certificate: true
              common_tls_context:
                tls_certificates:
                  - certificate_chain:
                      filename: /etc/envoy/certs/certificate.pem
                    private_key:
                      filename: /etc/envoy/certs/key.pem
                validation_context:
                  trusted_ca:
                    filename: /etc/envoy/certs/ca.pem
                  trust_chain_verification: VERIFY_TRUST_CHAIN

[arkodg]

thanks for raising this issue, this seems like mTLS support between downstream client and envoy which should definitely be supported by the project. Linking the Gateway API issue https://github.com/kubernetes-sigs/gateway-api/issues/91 which will be required to be able to describe this user intent.

[cpakulski]

Thanks for sharing that use-case. I think that the only trick here is delivery of self-signed CA certificate to the gateway. Gateway itself will push the certificate down to Envoy via SDS. Gateway may receive the certificate either encoded inline or from the file system.

[danehans]

After https://github.com/envoyproxy/gateway/pull/95 is merged, TLS config will need to be added.

[fubhy]

We do exactly what OP described (for the exact same reasons [to benefit from Cloudflares strong DDoS, bot detection, machine learning, etc.] for a custom gRPC API using Cloudflares Origin Pulls mTLS setup with the API (and our custom Envoy front proxy setup) hosted on fly.io. Good to see that others are doing similar setups and I'd love to see this as an Envoy Gateway example.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[danehans]

I think that the only trick here is delivery of self-signed CA certificate to the gateway.

https://github.com/envoyproxy/gateway/pull/208 adds control plane auth where certgen generates a self-signed CA and envoy-gateway/envoy cert/keys, then stores them as Kube secrets in the Envoy Gateway namespace.

[danehans]

Note that https://github.com/kubernetes-sigs/gateway-api/issues/91 has yet to be fixed and is still tagged as "help wanted".

[danehans]

@arkodg I suggest adding an agenda item for https://github.com/kubernetes-sigs/gateway-api/issues/91#issuecomment-1287437166 to the Gateway API community meeting. I expect a design doc, review, etc will be needed for the Gateway API community to achieve consensus.

[danehans]

Moving to the backlog since the dependent upstream issue, https://github.com/kubernetes-sigs/gateway-api/issues/91, is not fixed.

[arkodg]

since its unclear whether https://github.com/kubernetes-sigs/gateway-api/issues/91 will land before March 2024 when Envoy Gateway plans on going GA, plan on picking up this issue and adding this API field into ClientTrafficPolicy

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.