Traces with errors should not be sample rated

[avaly]

In order to make the best of the Error Tracking feature, we would like to ingest all errors generated by an application.

However, in a production environment where we use a low sample rate (e.g. DD_TRACE_SAMPLE_RATE=0.05), only a subset of traces are actually being sent from the agent upstream to Datadog. If we have errors thrown in the subset of traces which are not being sent, these errors never show up in Datadog UI.

I don't know if a change to enable this new behavior is required in this repository or in the agent repository. I'll leave that up to owners of these repositories to figure out the best implementation.

Steps to reproduce

Use the application code from #1944 including the workaround to attach errors to the service level span and run this code with the DD_TRACE_SAMPLE_RATE=0.01.

Expected behavior

All errors should be visible in the Error Tracking UI.

Actual behavior

Only 1% of the errors are visible in the Error Tracking UI.

[rochdev]

All traces are sent to the agent even if they are marked to be dropped. It's the agent that takes the final decision to keep or drop a span, so I would recommend opening an issue in https://github.com/DataDog/datadog-agent.

I would still leave this issue opened as we're planning to move that logic from the agent to each tracer, so later on we'll need to implement it in dd-trace as well.

[tlhunter]

@rochdev do you still want this issue to remain open?

[rochdev]

@tlhunter I think it's fine to close it as we'll end up doing whatever the agent does when the logic is moved. In the meantime, the issue belongs in the agent repository.