Add a flag to specify client certificate and private key that gRPC gateway provides to etcd server

[tatsuhiro-t]

What would you like to be added?

Add a flag to specify client certificate and private key that gRPC gateway provides to etcd server.

Why is this needed?

As https://github.com/etcd-io/etcd/issues/15755#issuecomment-1518536063 this comment says, currently gRPC gateway uses etcd server certificate as client certificate. This causes undesirable side effect and confusion that affect almost all etcd deployment. That is we need to add client auth key usage to sever certificate. In some deployments, we have separate CA for server certificate issuer and client certificate issuer. --trusted-ca-file specifies CA to verify client certificate, but because gRPC gateway uses server certificate as client certificate, we need to include both CAs so that it can verify client and server certificate.

It is best to provide a way to configure client certificate that gRPC uses so that the confusion and deployment issues stated above are fixed completely.

[oxxenix]

@tatsuhiro-t, @ahrtr hey! Can I work on this?

[ahrtr]

This feature is only useful when --client-cert-allowed-hostname is used to specify the allowed TLS hostname for client cert authentication, just as https://github.com/etcd-io/etcd/issues/15755 mentioned. I am not sure how many users are using --client-cert-allowed-hostname.

In some deployments, we have separate CA for server certificate issuer and client certificate issuer. --trusted-ca-file specifies CA to verify client certificate, but because gRPC gateway uses server certificate as client certificate, we need to include both CAs so that it can verify client and server certificate.

If we add the new flags you proposed, then it's still possible that users sign the certificate for grpc-gateway using a separate CA; in that case, we still need to include both CAs (one to verify the client, the other to verify the grpc-gateway).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[AdmiralNemo]

This feature is only useful when --client-cert-allowed-hostname is used to specify the allowed TLS hostname for client cert authentication, just as #15755 mentioned. I am not sure how many users are using --client-cert-allowed-hostname.

I don't think that's the case. Having the server certificate signed by a different CA than the one specified in --trusted-ca-file is enough to encounter issues like this:

{"level":"warn","ts":"2024-07-24T22:49:35.077771Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:46000","server-name":"","error":"tls: failed to verify certificate: x509: certificate signed by unknown authority"}
2024/07/24 22:49:35 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "0.0.0.0:2379", ServerName: "0.0.0.0:2379", }. Err: connection error: desc = "error reading server preface: remote error: tls: unknown certificate authority"

It should be possible for the server certificate to be signed by a different CA than the client certificates, as this is a fairly common pattern when using mTLS, in my experience.

[lllamnyp]

This feature is only useful when --client-cert-allowed-hostname is used to specify the allowed TLS hostname for client cert authentication, just as #15755 mentioned. I am not sure how many users are using --client-cert-allowed-hostname.

Besides the failure mode mentioned by @AdmiralNemo , I also ran into several warnings when the server cert did not specify client_auth as one of the key usages:

{"level":"warn","ts":"2024-08-13T09:09:47.232813Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"10.129.217.153:49124","server-name":"","error":"tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"}
{"level":"info","ts":"2024-08-13T09:09:47.233106Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Creating new client transport to \"{Addr: \\\"10.129.217.153:2379\\\", ServerName: \\\"10.129.217.153:2379\\\", }\": connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.