opentelemetry-ruby icon indicating copy to clipboard operation
opentelemetry-ruby copied to clipboard

Added GRPC example

Open pinglamb opened this issue 4 years ago • 8 comments

Hi, I have added a basic GRPC example (#96).

I took reference of the Java and Go example and used GRPC::ServerInterceptor for injecting the trace logic. Other parts are mostly copied from Ruby GRPC example (with some modifications), including the .proto file and server/client implementation.

I have 2 parts I am not sure about, which would like to seek some suggestions/guidance:

  • How the parent context should be extracted? I saw in the http example it uses OpenTelemetry.propagation.http.extract(env), should I add grpc util or just extract it from call.metadata in the Interceptor. Also, what metadata should I extract from there?
  • Should I assign the grpc status to span status when error happens?

I found that protobuf seems to have issue with Ruby 2.7 (https://github.com/protocolbuffers/protobuf/issues/7070). I will test further see if it is my configuration or protobuf issue.

Any feedback is welcome, it is my first time contribution. Thank you very much.

pinglamb avatar Apr 22 '20 11:04 pinglamb

Thanks for working on this!

The protobuf issue with Ruby 2.7 has been a blocker for us (Shopify) too. I’ll check internally to see if and how we’re working around it.

Assigning grpc status to span status is required: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/trace/semantic_conventions/rpc.md#status

The semantic conventions for gRPC are documented here: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/trace/semantic_conventions/rpc.md#grpc

The specification doesn’t describe how propagation is supposed to work with gRPC, but the Go implementation simply sets the keys in the metadata. I think we want to use the text propagators rather than http for that, since the latter (for Ruby) expects Rack formatted keys. @mwear can definitely correct me on that.

fbogsany avatar Apr 22 '20 21:04 fbogsany

Thanks @fbogsany for pointing out the spec.

For the span name, as the grpc lib will underscore the rpc method name (https://github.com/grpc/grpc/blob/v1.28.1/src/ruby/lib/grpc/generic/service.rb#L98), we have to camelize it back.

For the peer attributes, we only have a string containing both the ip address and port so I added a regex to parse it.

Here is an example of current span data:

#<struct OpenTelemetry::SDK::Trace::SpanData
 name="helloworld.Greeter/SayHello",
 kind=:server,
 status=0,
 parent_span_id="0000000000000000",
 child_count=0,
 total_recorded_attributes=4,
 total_recorded_events=0,
 total_recorded_links=0,
 start_timestamp=2020-04-23 19:42:36 +0800,
 end_timestamp=2020-04-23 19:42:36 +0800,
 attributes=
  {"component"=>"grpc",
   "rpc.service"=>"Greeter",
   "net.peer.ip"=>"::1",
   "net.peer.port"=>"60369"},
 links=nil,
 events=nil,
 library_resource=
  #<OpenTelemetry::SDK::Resources::Resource:0x00007fb1eab52680
   @labels={"name"=>"grpc", "version"=>"semver:1.0"}>,
 span_id="f8f0633f420eafa1",
 trace_id="4cf5dee9d1d02d7242123d2fed2acd27",
 trace_flags=#<OpenTelemetry::Trace::TraceFlags:0x00007fb1eb0e70c8 @flags=1>>

pinglamb avatar Apr 23 '20 11:04 pinglamb

I also assigned the context to be OpenTelemetry.propagation.text.extract(call.metadata), not sure if I am doing it right.

pinglamb avatar Apr 23 '20 12:04 pinglamb

I have studied the Go implementation again and did some adjustments:

  • opentelemetry-go made the grpctrace a plugin (https://github.com/open-telemetry/opentelemetry-go/tree/master/plugin/grpctrace) which contains the server/client interceptors for different scenarios. I copied the idea and the interceptors are grouped under a single module GRPCTrace under the example lib folder which can potentially be extracted out in the future
  • Understand better about the context propagation, used the OpenTelemetry.propagation.http.inject and OpenTelemetry.propagation.http.extract for the propagation in the interceptors (referencing Go implementation again)

Two hacks are needed so far because the grpc interceptors does not have enough info exposed (it is still an experimental API) for building the span:

  • For client interceptor, it can't access the peer info from the call so instance_variable_get needs to be used to access underlying data
  • For server interceptor, the RPC method name is lost (which is available for client interceptor). Right now, I try to reconstruct it from the actual method object

I will continue to explore for StreamServer and StreamClient

pinglamb avatar May 08 '20 11:05 pinglamb

@pinglamb are you still working on this?

fbogsany avatar Sep 21 '20 19:09 fbogsany

I can continue to work on this, any comments so far? Thanks!

pinglamb avatar Sep 22 '20 05:09 pinglamb

@pinglamb have any progress with this one?

Asafb26 avatar Nov 01 '20 12:11 Asafb26