libdatadog
libdatadog copied to clipboard
[Data-Pipeline] Add stats computation
What does this PR do?
Work from @paullegranddc in #458
- Add
stats_exporterto receive, aggregate and send span stats - Move try_c to
ddcommon_ffi
This is a first implementation which still have some caveats:
- We only store one time bucket and all SpanStats received are aggregated in this bucket, this requires the client to call
sendregularly to avoid a time bucket too wide. - We don't aggregate on
peer.tags - We currently have two different exporter one for sampled spans and another for stats. The sampled spans need to be sent to both. To merge both exporter we need
- a way to send partial spans for the unsampled spans, to avoid serialization of fields not used for stats computation
- replace the synchrone send of the spans exporter, to be able to aggregate the stats on time interval chosen by libdatadog instead of the client span submission rate.
Motivation
Moving stats computation to libdatadog would allow all tracers to compute stats locally and only send sampled spans to the agent. This will reduce traffic to the agent and avoid serialization of data which are going to be dropped in the agent
Drawbacks
The current implementation doesn't reduce the amount of serialization since we need to serialize spans to send them from the tracer to libdatadog.
Additional Notes
Anything else we should know when reviewing?
How to test the change?
You can modify and run the example to send stats to a test agent
Codecov Report
Attention: Patch coverage is 70.38540% with 146 lines in your changes missing coverage. Please review.
Project coverage is 70.50%. Comparing base (
f380cb5) to head (d156de3).
Additional details and impacted files
@@ Coverage Diff @@
## main #507 +/- ##
==========================================
+ Coverage 70.21% 70.50% +0.28%
==========================================
Files 206 208 +2
Lines 27815 28254 +439
==========================================
+ Hits 19530 19920 +390
- Misses 8285 8334 +49
| Components | Coverage Δ | |
|---|---|---|
| crashtracker | 16.86% <ø> (ø) |
|
| datadog-alloc | 98.73% <ø> (ø) |
|
| data-pipeline | 65.45% <76.04%> (+14.30%) |
:arrow_up: |
| data-pipeline-ffi | 0.00% <0.00%> (ø) |
|
| ddcommon | 86.26% <0.00%> (-0.15%) |
:arrow_down: |
| ddcommon-ffi | 75.31% <ø> (ø) |
|
| ddtelemetry | 60.50% <11.11%> (+1.47%) |
:arrow_up: |
| ipc | 84.13% <ø> (ø) |
|
| profiling | 78.68% <ø> (ø) |
|
| profiling-ffi | 58.26% <ø> (ø) |
|
| serverless | 0.00% <ø> (ø) |
|
| sidecar | 35.61% <0.00%> (-0.03%) |
:arrow_down: |
| sidecar-ffi | 0.00% <0.00%> (ø) |
|
| spawn-worker | 54.98% <ø> (ø) |
|
| trace-mini-agent | 70.93% <ø> (ø) |
|
| trace-normalization | 98.24% <ø> (ø) |
|
| trace-obfuscation | 95.73% <ø> (ø) |
|
| trace-protobuf | 77.16% <ø> (ø) |
|
| trace-utils | 90.68% <ø> (+0.77%) |
:arrow_up: |
Benchmarks
This comment was omitted because it was over 65536 characters.Please check the Gitlab Job logs to see its output.