gRpc service root span resource name overwritten when making JaxRs.request client call

[berksean]

I've been observing some odd behavior in APM where gRpc server requests (operation:grpc.server) are not showing up under the expected resource name (e.g. foo.package.Service/Method), and instead appear with the resource name associated with on outgoing client call made while handling the gRpc request (e.g. operation:jax_rs.request, GET /some/path).

After running the service/agent locally and debugging the service, it appears that the root span's resource name gets overwritten as part of the JaxRsAnnotationsDecorator.onJaxRsSpan processing here. It appears to be setting the resource name on the local root span, which seems incorrect.

It does look like the resource name overwrite behavior of HttpResourceDecorator.withRoute code is configurable via dd.http.server.route-based-naming, and this code seems to be the only reader of that config. Setting the config value to false seems to correct the behavior, but as I couldn't find clear documentation on the purpose of the flag, I figured I'd raise the issue here. It's not clear to me that things are functioning as expected, and I'm not sure what the implications of turning that configuration off might be.

[bantonsson]

Hi @berksean thank you for reporting this and the thorough analysis. It looks from what you're describing that your code is using the jax-rs server annotations for outgoing requests as well, is that correct? If that is the case then the current jax-rs instrumentation will also treat them wrong and tag them as server spans. Is this something that you have seen in your dashboards?

[berksean]

Oddly not. It seems like other services using a similar client don't run into this issue, though most other services are not fronted by gRpc server handler. Is it possible the priority component is coming into play? Prior to being overwritten, I notice that the gRpc resource has priority 0, while I believe the overwriting resource name is 3 (HTTP_FRAMEWORK_ROUTE).

In terms of usage of jax-rs server annotations, that's probably an artifact of how the clients are generated. We're using a client artifact provided by the service that uses Feign to build a client off the annotated service interface. Not sure if that's super non-standard/unexpected.

[berksean]

Also @bantonsson - Just re-read your question and I see that you're asking if the APM service overviews are marking the requests as server spans. I'm actually not familiar with how to determine that. Please let me know what I should look for and I'll get back to you. I will mention that while debugging, there were two spans associated with the GET /some/path resource. The root span and a span created as part of the client call. The root span's resource name got overwritten as part of the client span's creation.

Here's an image of the resources the APM service overview lists for the service:

Qualitatively, under the grpc.server operation is see resources like GET /a/b/c, and I also see resources for GET /a/b/c under the jax_rs.request operation. I do not see anything that looks like a servlet.request operation associated with the resource. Not sure if that answers your question.

[berksean]

Slight correction to the above. The GET /a/b/c resource doesn't appear under the jax_rs.request operation. The jax_rs.request operation has an InterfaceClass.actionMethod type resource. The http.request operation has the GET /a/b/c resource.

[jakubkalicki-tink]

Hi @bantonsson,

I have the same issue. Is there any solution / workaround for that? Should i simply set DD_HTTP_SERVER_ROUTE_BASED_NAMING to false?