dd-trace-rb
dd-trace-rb copied to clipboard
DataDog
Questions about Asynchronous Tracing
TL: DR How do I link traces that function asynchronously with dd-trace?
Long Version
I am looking for a way to improve QUE gem instrumentation. The problem I am facing at the moment is about linking traces of background jobs inserted and processed off of it's storage mechanism (PostgreSQL).
I would like to see a continuity of trace spans for jobs that get put onto a queue and when they get processed so that I can view a trace's entire lifecycle.
Que works very similar to DelayedJobs where it's client pushes jobs to a database and processes them by pulling jobs off of that database.
The existing bits of documentation about Asynchronous tracing and other instrumentation codebases didn't help me much. I was thinking of persisting the trace_id alongside jobs in the db and when processing them somehow link spans with that trace_id. I am on the right path? Any examples of code that could show me how this is accomplished in other places?
👋 @hs-bguven, the functionality that you are looking for is an implementation of the FollowsFrom/ChildOf mechanism.
We have it in our roadmap as one of the most requested features, so we area definitely working on it internally, but there is still some work to be done to ensure our dashboards can probably report these spans.
Until then, your option is to add additional tracking information to your job arguments and use those to set custom tags in the job executor thread, which I acknowledge is not ideal.
@marcotc Thank you for getting back to me.
I am happy that it's on your roadmap, no wonder its a frequently requested feature : )
set custom tags in the job executor thread, which I acknowledge is not ideal.
Would you be able to point me in the right direction with examples/code-samples? Again I am more than happy to take it from there and work on those improvements myself. I would be interested in having a solution in place until you guys work on the delivery of the actual feature.
Would you be able to point me in the right direction with examples/code-samples? Again I am more than happy to take it from there and work on those improvements myself. I would be interested in having a solution in place until you guys work on the delivery of the actual feature.
My non-ideal suggestion for the time being is adding a unique tag to both the originating context as well as to the worker context:
def handle_web_request
Worker.enqueue(arg1, arg2, request_id: request_id) # request_id is any unique identifier for this specific request your application can provide, e.g.:
Datadog.tracer.active_span.set_tag("request_id", request_id.to_s) if Datadog.tracer.active_span
end
class Worker < Que::Job
def run(arg1, arg2, request_id: nil)
Datadog.tracer.active_span.set_tag("request_id", request_id.to_s) if Datadog.tracer.active_span
# ...
end
end
Your originating context might already have a unique value attached to that endpoint, so you might not need to tag it at that point.
There's of course the severe limitation of not being able to easily navigate between these two traces, which is what our longer term plan addresses.
