Support custom trace propagation format in ocgrpc

[ashi009]

The default ocgrpc handler assumes all incoming trace span context will be passed in binary format using grpc-trace-bin key in metadata.

In many cases, the calling application may use a different trace propagation format, eg. x-cloud-trace-context for GCP.

There should be a way to add additional support to other propagation formats.

A proposed approach is that:

• add a new interface in trace/propagation package:

  type GRPCFormat interface {
    SpanContextFromMetadata(metadata.MD) (sc trace.SpanContext, ok bool)
    SpanContextToMetadata(trace.SpanContext, metadata.MD)
  }
  

• add option to ocgrpc.ServerHandler to specify the parent span extraction preference

• add option to ocgrpc.ClientHandler to specify the span propagation preference

[semistrict]

/cc @rakyll

[ashi009]

Context: https://github.com/cloudendpoints/esp/issues/416

[odeke-em]

Thank you for filing this issue @ashi009 and welcome to OpenCensus, sorry for the radio silence too. Perhaps this issue is a superset of https://github.com/census-instrumentation/opencensus-go/issues/666?

[ashi009]

@odeke-em Apparently yes. Though the issues are from different backgrounds, the solution is actually the same.

[rakyll]

/cc @bogdandrutu to decide on this topic.

[vvakame]

Hi, I & My company have interesting in this topics. now, We are using DataDog library in our services. not OpenCensus. But I want to start using OpenCensus.

My plan is

• use OpenCensus + DataDog exporter in some services gradually
  • We need to use DataDog's propagation format
• use OpenCensus + DataDog exporter in all services!
  • We can switch into grpc-trace-bin
  • We need to be able to accept both format for a while
• We can choose favorite exporter!
  • e.g. DataDog + StackDriver Tracing

Can I implement and send a pull request?

[vvakame]

@bogdandrutu can we talk about this topic?

[odeke-em]

Hello @vvakame, on a side track about what you've mentioned in https://github.com/census-instrumentation/opencensus-go/issues/838#issuecomment-460975288

We've been working on the OpenCensus Agent https://opencensus.io/agent https://github.com/census-instrumentation/opencensus-service and given that you'd like to switch to using OpenCensus, the Agent will soon support receiving traffic that was intended for Datadog and other backends and then you can export it out whichever way. I am haven't yet worked on the Datadog receiver, but if you are interested and enthusiastic about it, I can make this a top priority and work on shipping out the Datadog receiver. Please feel free to instead migrate this specific conversation to https://github.com/census-instrumentation/opencensus-service/issues/358 and https://github.com/census-instrumentation/opencensus-service/issues/384.

Thank you.

[vvakame]

thanks. We don't use OpenCensus agent yet. I'll read that document and make conversation with my co-worker.

[vvakame]

@odeke-em I have a question. OpenCensus agent is agent, not proxy. (is it right?)

Our microservices will be mixed about use OpenCensus agent or not(-yet). I want to know that Can OpenCensus agent works fine under mixed environment?

for example client: A microservice(use OC agent) server: B microservice(use DataDog only) those services communicate by gRPC. If A called B's method. OC agent & collector can't construct single trace. because OC agent can't modify gRPC's metadata. can't make relation between A's span to B's span on DataDog.

[someone1]

I seemed to have stumbled across another scenario on GCP that this would help to fix. My setup: CloudRun runs ESP for a gRPC backend I have in a Managed Instance Group on GCE. The request flow is:

Cloud Run (w/ ESP) -> HTTPS Load Balancer -> Cloud IAP -> Managed Instance Group Backend Service (GCE VM)

Cloud Run and the Load Balancer seem to log using the same trace ID, however, the Managed Instance group doesn't get the trace in the correct grpc-trace-bin metadata key, instead, it receives a x-cloud-trace-context metadata header instead. Assuming https://github.com/cloudendpoints/esp/pull/503 fixed the issue with ESP being the culprit, and the HTTPS Load Balancer does indeed work correctly as mentioned earlier in this issue, then it seems Cloud IAP is to blame (I have no visibility into Cloud IAP so I cannot confirm). My Backend service ends up generating a new trace ID for the request instead of pulling from the Stackdriver HTTP metadata key/format.