libdatadog icon indicating copy to clipboard operation
libdatadog copied to clipboard

[Data-Pipeline] Add stats computation

Open VianneyRuhlmann opened this issue 1 year ago • 2 comments

What does this PR do?

Work from @paullegranddc in #458

  • Add stats_exporter to receive, aggregate and send span stats
  • Move try_c to ddcommon_ffi

This is a first implementation which still have some caveats:

  • We only store one time bucket and all SpanStats received are aggregated in this bucket, this requires the client to call send regularly to avoid a time bucket too wide.
  • We don't aggregate on peer.tags
  • We currently have two different exporter one for sampled spans and another for stats. The sampled spans need to be sent to both. To merge both exporter we need
    1. a way to send partial spans for the unsampled spans, to avoid serialization of fields not used for stats computation
    2. replace the synchrone send of the spans exporter, to be able to aggregate the stats on time interval chosen by libdatadog instead of the client span submission rate.

Motivation

Moving stats computation to libdatadog would allow all tracers to compute stats locally and only send sampled spans to the agent. This will reduce traffic to the agent and avoid serialization of data which are going to be dropped in the agent

Drawbacks

The current implementation doesn't reduce the amount of serialization since we need to serialize spans to send them from the tracer to libdatadog.

Additional Notes

Anything else we should know when reviewing?

How to test the change?

You can modify and run the example to send stats to a test agent

VianneyRuhlmann avatar Jun 26 '24 13:06 VianneyRuhlmann

Codecov Report

Attention: Patch coverage is 70.38540% with 146 lines in your changes missing coverage. Please review.

Project coverage is 70.50%. Comparing base (f380cb5) to head (d156de3).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #507      +/-   ##
==========================================
+ Coverage   70.21%   70.50%   +0.28%     
==========================================
  Files         206      208       +2     
  Lines       27815    28254     +439     
==========================================
+ Hits        19530    19920     +390     
- Misses       8285     8334      +49     
Components Coverage Δ
crashtracker 16.86% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 65.45% <76.04%> (+14.30%) :arrow_up:
data-pipeline-ffi 0.00% <0.00%> (ø)
ddcommon 86.26% <0.00%> (-0.15%) :arrow_down:
ddcommon-ffi 75.31% <ø> (ø)
ddtelemetry 60.50% <11.11%> (+1.47%) :arrow_up:
ipc 84.13% <ø> (ø)
profiling 78.68% <ø> (ø)
profiling-ffi 58.26% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 35.61% <0.00%> (-0.03%) :arrow_down:
sidecar-ffi 0.00% <0.00%> (ø)
spawn-worker 54.98% <ø> (ø)
trace-mini-agent 70.93% <ø> (ø)
trace-normalization 98.24% <ø> (ø)
trace-obfuscation 95.73% <ø> (ø)
trace-protobuf 77.16% <ø> (ø)
trace-utils 90.68% <ø> (+0.77%) :arrow_up:

codecov-commenter avatar Jul 05 '24 14:07 codecov-commenter

Benchmarks

This comment was omitted because it was over 65536 characters.Please check the Gitlab Job logs to see its output.

pr-commenter[bot] avatar Jul 05 '24 15:07 pr-commenter[bot]