Evaluate & potentially replace `https://github.com/soheilhy/cmux`

[ahrtr]

What would you like to be added?

etcdserver depends on https://github.com/soheilhy/cmux to receive both gRPC and REST connections on the same TCP listener/port. But the library isn't well maintained; note it hasn't been updated for more than 3 years at the time of writing this ticket. It also has issue working with HTTP/2, please refer to soheilhy/cmux/issues.

We should evaluate & summarize the problems the library has, and

• either try to find an alternative solution.
• or completely separate gRPC and REST/HTTP connections;

I expect someone with more network expertise could drive this effort.

Why is this needed?

See above

[serathius]

cc @aojea

[fuweid]

The ServeHTTP is still experimental feature in gRPC side https://pkg.go.dev/google.golang.org/grpc#Server.ServeHTTP. There is different between gRPC and net/http about the underlying implementation of HTTP2 transport. As far as I know, the gRPC implementation has some auto-tune features on window-update. If there is good performance provided gRPC, I think we can consider using separate connections.

[serathius]

I think we can consider using separate connections.

Using separate connections is much better as shown by https://github.com/etcd-io/etcd/issues/15402, however removing connection multiplexing would be a breaking change for users. If we look at the list of endpoints in https://github.com/etcd-io/etcd/issues/15402#issuecomment-1460061395, the most impacted would be would be:

• Users of /health via health probes, so I assume most of the users.
• Users of /v3/* v3 grpc gateway, which based on the questionnaire we sent are more than we expected as not every programming language supports grpc.
• Users of /metrics that didn't separate the port via --listen-metrics-urls.

[rleungx]

We encountered this issue https://github.com/grpc/grpc-go/issues/7261 caused by serverHandlerTransport.

[serathius]

@rleungx Looked into the problem. Doesn't look it affects etcd as we don't allow arbitrary as we don't use stream requests with large amounts of data.

[rleungx]

@rleungx Looked into the problem. Doesn't look it affects etcd as we don't allow arbitrary as we don't use stream requests with large amounts of data.

We are using the embedded etcd to serve HTTP and gRPC calls, the latter includes both unary and streaming requests.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.