Load certificate from SDS server

[freelich-du]

Envoy supports loading certificate from user provided SDS server. https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret.html?highlight=sds While Contour doesn't support it yet. We need this feature to achieve better certificate management and reduce security risk.

[youngnick]

Thanks for this @freelich-du.

Contour already uses SDS to supply Kubernetes secrets to Envoy, and I'm unsure if Envoy can have multiple SDS servers defined. Perhaps @wrowe or @sunjayBhatia have some insight there?

What are you aiming to achieve using SDS? Kubernetes Secrets already allow quite dynamic certificate management, and Contour will automatically update Envoy's secrets with the contents of relevant Kubernetes Secrets.

[loveyana]

@youngnick We are investigating this project and we want to use it as kubernetes ingress controller. We want my ingress to use the MTLS certificate issued under Spire, they support SDS provider for Envoy. so we have the same need to support Envoy SDS, I am not sure if contour should support this feature.

We did not adopt a service mesh solution, like Istio, mainly because we felt it would be intrusive for the application

[sunjayBhatia]

SDS secret config is often a list so you could specify multiple sources of secrets, e.g. in the extensions.transport_sockets.tls.v3.CommonTlsContext: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#envoy-v3-api-field-extensions-transport-sockets-tls-v3-commontlscontext-tls-certificate-sds-secret-configs

I'm curious how Spire would fit into how we configure TLS secrets today, maybe something like:

• add support for an SDS provider as an ExtensionService feature, or other mechanism to let Contour know how to tell Envoy to contact Spire and so Contour can validate this should be valid
  • from a quick look on the example Spire deployment in k8s, it runs as a DaemonSet and Envoy would communicate over a UDS?
  • Would have to keep this mechanism open to multiple deployment mechanisms for Spire?
• when configuring TLS on HTTPProxy you can specify a k8s secret and/or (choice of and or or is important) say you want to use the Spire cluster/server configured before
• Contour would setup Spire as a SDS cluster and configure it in the tls config for an upstream/downstream

[loveyana]

@sunjayBhatia I think the first case has the least changes to the project，envoy currently supports grpc calls to spire SDS provider using the UDS connection of spire-agent and can verify the peer end spiffeID，its configuration is roughly as follows:

transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
    common_tls_context:
      tls_certificate_sds_secret_configs:
      - name: "spiffe://domain.test/upstream-workload"
        sds_config:
          resource_api_version: V3
          api_config_source:
            api_type: GRPC
            transport_api_version: V3
            grpc_services:
              envoy_grpc:
                cluster_name: spire_agent
      combined_validation_context:
        default_validation_context:
          match_subject_alt_names:
            exact: "spiffe://domain.test/downstream-workload"
        validation_context_sds_secret_config:
          name: "spiffe://domain.test"
          sds_config:
            resource_api_version: V3
            api_config_source:
              api_type: GRPC
              transport_api_version: V3
              grpc_services:
                envoy_grpc:
                  cluster_name: spire_agent
      tls_params:
        ecdh_curves:
          - X25519:P-256:P-521:P-384

And If contour supports a configuration similar to SDS above for ingress, we might offer an Entry permission to envoy Pods to obtain additional service certificates.

[youngnick]

@loveyana, I'm sorry, but I don't understand why you don't want to use Kubernetes Secrets as the mechanism here. Contour already has complete support for referencing Kubernetes Secrets in any configuration you could require. Why do you need an additional method to send secrets to Envoy? Why not have Spire create Kubernetes Secrets instead of bypassing Kubernetes?

Contour is, by design, a Kubernetes controller, and we have spent a lot of time thinking about safely referencing Secrets. I'm wary of doing a similar amount of work to expose secrets by some other method.

[loveyana]

@youngnick thank you very much for your reply, I am not very familiar with the contour project, so some ideas may not fit the design of the community, but I can share our scene

We have an idea about ingress connecting to spire-agent to obtain its own x509 certificate. It is mainly based on the following business scenarios. We use spire as a zero-trust infrastructure in our company on a large scale. It provides SDK calls and envoy sidecars. The mode is used as a calling method to obtain an SVID (including a certificate of spiffeID information) for a service to use for mtls connection with other services. In one of our scenarios, our api gateway is under another kubernetes cluster, and it will call the service under our cluster. We hope to expose this service port through ingress and ensure the mtls communication from the api gateway to the port through ingress.

[youngnick]

So, the connection to SPIRE is for the Envoy that Contour is running to get an mTLS client certificate, that it will use to identify itself to your internal infrastructure, not to use SPIRE to issue TLS certificates for the external domain names referenced by Ingress config?

[loveyana]

@youngnick It’s not. Maybe I didn’t explain too clearly. We need to use spire to issue TLS certificates for the external domain names referenced by the Ingress configuration. Because apigateway is outside the cluster, we let it call spire to obtain its mtls certificate and access back-end service cluster ingress, we want to use the Envoy that Contour is running to check the certificate and spiffeID it carries.

[youngnick]

Thanks @loveyana, I think it's important for us to be super clear on this one!

So, I think what you're saying is that the flow works like this:

• Spire is running somewhere, it issues TLS certificates for external domain names.
• You have an API Gateway outside the cluster that you want to connect to services inside the cluster (so you need a Ingress-style config to translate between outside-cluster things and inside-cluster things)
• You want the Ingress config to use the certificates generated by Spire.

Hopefully that's right.

If that is right, I still feel like the best thing to do would be to find a way to have Spire create Kubernetes Secrets automatically. It seems weird that it can't do this already, to be honest, given that it's designed to be used with Kubernetes. Then those Secrets could easily be used by Ingress or HTTPProxy objects, and Contour will handle passing them to Envoy when they are created or updated by the Spire integration.

If we did give the ability to configure another SDS server, the problem is that Contour needs to generate HTTPConnectionManagers for Envoy that reference the secrets in the Spire SDS by name, but Contour doesn't have a way to know the names Spire is using for the secrets in its SDS without also connecting to Spire itself. Without having a way for Contour to know the correct SDS secret name, I don't see how it would work.

So, I guess what I'm saying is, I understand what you're asking for, and it seems reasonable, but I don't feel confident that if we built it, it would actually give you the functionality you want. I strongly recommend trying to see if you can get Spire to write out the generated TLS certificates as Kubernetes Secrets instead.

[evan2645]

If we did give the ability to configure another SDS server, the problem is that Contour needs to generate HTTPConnectionManagers for Envoy that reference the secrets in the Spire SDS by name, but Contour doesn't have a way to know the names Spire is using for the secrets in its SDS without also connecting to Spire itself. Without having a way for Contour to know the correct SDS secret name, I don't see how it would work.

Sorry for digging up the stale issue, but it continues to pop up. Thought I'd drop a quick note here that SPIRE can be configured to use a static SDS secret name (in fact, I believe it does this by default), which could then also be configured on Contour.

[tsaarni]

Is https://spiffe.io/docs/latest/microservices/envoy/ targeting the service mesh use case, where each service is co-located with their own "sidecar" instance of Envoy? Each service gets only single workload identity and therefore the local Envoy fetches just single SDS secret, served by the local SPIRE agent by the default SDS name default_svid_name. Is that how it is used?

How would that map to a use case where Envoy acts as an ingress proxy, and it should be able to act as server on behalf of several virtual hosts / domains (according to the HTTPProxy definitions), each with their own identities/server certificates? Like @loveyana wrote:

We need to use spire to issue TLS certificates for the external domain names referenced by the Ingress configuration.

In the service mesh use case, a single certificate may be used by the sidecar Envoy as both server certificate when accepting inbound TLS connections from the mesh, as well as client certificate when initiating outbound connections to the peers in the mesh. In ingress use case Envoy is expected to serve inbound connections from external clients with several server certificates. However, when it then routes the requests to upstream, towards backend services within the cluster, currently we only allow single client certificate for the upstream connections. Would that upstream use case map better with the certificate that SPIRE agent serves by the default SDS name?