Propagate current trace context to Active Record async query threads
What does this PR do?
The Active Record integration (for anything greater than v7.0.0) will now wrap ConnectionPool#build_async_executor with the already-existing ContextCompositeExecutorService from the concurrent-ruby tracer integration, ensuring that the current trace is continued within the thread.
Closes #3465.
Motivation:
When leveraging Active Record's async query functionality (load_async, async_count, etc.), the queries are executed in a different thread, via concurrent-ruby's ThreadPoolExecutor. This causes the span for that query to not show up in the parent trace (often a rack.request).
How to test the change?
I included tests here, but am definitely open to other approaches here!
@marcotc was just wondering if this was something you'd be able to take a look at when you get a chance. It would be great to get this into a release soon! 🤞🏻
@agrobbin, yes! It's pretty much what I've been doing in the last two weeks 😅: https://github.com/DataDog/dd-trace-rb/pull/4872
It required quite a major change in how we flush traces and track background threads (which is a bug that greatly affects the concurrent-ruby instrumentation). More specifically, this didn't work correctly: https://github.com/DataDog/dd-trace-rb/blob/44947b27e86b871e6028d5bece5087a145cabcb5/spec/datadog/tracing/tracer_spec.rb#L993-L1001 when span-1 finished the whole wrapping trace would finish, and span-2 would create a new trace as a fallback that was not associated with the digest.
This affects async queries because multiple queries can run in the background for a single load_async, depending on the DB adapter, and those would create new, unrelated traces in the background.
I still need to to fix all failing tests, and add new tests to ensure I didn't break basic span flushing.
Ohh, wow, way more complicated than I thought! Looking forward to seeing all of that work land. 🎉