apm-agent-rum does not set the custom transaction as root

[suchcodemuchwow]

In our kibana performance tests, we are creating a custom transaction for each journey and controlling the headless browser on our end. From the beginning of the first request, we are intercepting all browser requests and modifying headers by adding traceparent from the transaction we created for the whole journey. So basically our process looks like this:

Test Runner(FTR) ----> Chromium(playwright,apm-rum-agent) -----> Kibana Server(apm-node-agent)

What we discovered is that even though we started transactions from FTR side browser acts in the way that it's the root of the trace. And If we look at FTR service as a result we get this:

Here we cannot see the frontend transaction inside the service we created.

And inside of kibana-frontend service, we can see the transactions (steps) but the order is messed up.

And from the logs we can also see that all frontend and backend has same trace-id: 9e1f4c373e615f6010b6648b85012396

For implementation details: https://github.com/elastic/kibana/pull/124259/files#diff-3225cc5c1861df1fad69fa164daff4dac70d3091e9cf6d4f3b8233590cffa923R181

cc @spalger @mshustov

[spalger]

Talked with @vigneshshanmugam about this yesterday and it sounds like what we need is a feature that will allow us to record RUM transactions as children of other transactions. The features we're using today assign the server-side transaction that loaded the page as a span of the RUM transaction, what we want instead is the ability to provide trace parent information to the RUM agent. Because of that I'm going to change this from bug to :New Feature. That being said, this is pretty important to both Synthetics and Kibana performance debugging so I hope that we can get this feature prioritized.

[vigneshshanmugam]

I spoke with @devcorpio about this issue and gave some details around reproducing this behaviour with Synthetics which would be the same as Kibana FTR - https://github.com/elastic/synthetics/pull/460. We will try to get this sorted out quick to unblock the Kibana team. Thanks both.

[paulb-elastic]

Relates to https://github.com/elastic/uptime-dev/issues/86

[lizozom]

Hey @paulb-elastic any updates on this?

[devcorpio]

Hi @lizozom,

We have been trying to accomplish this via https://github.com/elastic/synthetics-dev/issues/86

Unfortunately, the first approach we followed did not end well, yet.

After that, we focused on the crosslinking thing as a "separate feature" and we didn't give this enough priority. Here you can see the approach we tried to follow (crosslinking from synthetics UI to APM ui)

However, that approach had a few drawbacks, too (we will update the linked ticket soon explaining that)

Now, there is a third approach (that would also help to accomplish what we want to do here) that we want to try and it is:

We want RUM to check if Synthetics (or another runner) has injected a specific piece of data in the html document, for instance:

window.elasticSynthetics = { monitorId: "123" stepId: "234" transactionId: "6567" ... }

We need to propagate the data in that way, because the browsers doesn't allow you to read the headers of a loaded page via JavaScript.

With that information we will be able to addGlobal labels that will allow us to identify that certain transactions correspond to certain synthetic monitors and steps, for instance. Also we could somehow adapt the PR of the first approach to use the transactionId of the elasticSynthetics as the parent of the page load transaction.

@vigneshshanmugam @kyungeunni would you like to add something here?

Cheers, Alberto