Investigate options around latency reporting and comparison with server

[cretz]

Is your feature request related to a problem? Please describe.

We are seeing some differences in client request latency and server request latency. This is to investigate ways to track that and find out why.

Describe the solution you'd like

We may want to try two things:

• Seeing if we can have the server have a gRPC interceptor that puts a start-to-end duration in gRPC metadata and have a number/percent at which a difference between that and the client duration are large enough to at least trigger a warning
• See if there is a gRPC metric that is low-level enough to apply to HTTP2 messages specifically so that we can see if there is a difference between our interceptor-based ones (that surround serialization and other operations) and the low-level ones

[cretz]

It appears that https://pkg.go.dev/google.golang.org/grpc/stats is available to provide lower-level stats, I will confirm how it is implemented in gRPC itself and then look to expose that info.

[cretz]

gRPC stats do happen after interceptors. I have made an example of using gRPC stats at https://github.com/temporalio/samples-go/compare/master...cretz:grpc-stats that require dial options exposed in https://github.com/temporalio/sdk-go/compare/master...cretz:grpc-options.

Will wait for feedback/updates on how to proceed.