What does this PR do?

This PR re-introduces the Zstd compression initially implemented on #23806 The original PR surfaced critical build issues on environments where cgo is not available, therefore it had to be rolled back.

~We now add a new build tag, zstd, to determine whether Zstd is available or not, and fall back to gzip compression otherwise.~

To work around OTel limitations, we have now created a new trace/compression component, with 2 implementations:

impl-gzip: existing gzip compression algorithm, it represents no change compared to current compression logic.
impl-zstd: Zstandard compression, requires cgo.

This component also defines 3 new Go modules, in the /def, /impl-gzip, and /impl-zstd. This is requested by OTel, and goes in accordance with the Components documentation

To enable Zstd, zstd-encoding configuration setting must be passed (intentionally undocumented). When this setting is pased, trace-agent will use zstd. OTel usages are configured to always use gzip, and never import impl-zstd (this preventing build issues when cgo is not available).

Motivation

From the original PR #23806

Investigations showed that we can have better compression of our payloads without performance impacts. This can lead to significant data transfer cost savings on our end, and cost savings for our customers as well.

Additional Notes

From the original PR #23806

More information on the initial investigation in this doc

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Scenario 1:

Without changes, sending traces should still work out of the box. Encoding type should be gzip.

Scenario 2:

Add zstd-encoding in DD_APM_FEATURES Sending traces work. Encoding type should be zstd CPU usage shouldn't be higher than in the 1st scenario.

Jun 17 '24 13:06 ichinaski

Benchmarks

Benchmark execution time: 2024-06-17 13:14:33

Comparing candidate commit 500367f8dc363836fa342a1825ffb971529e7431 in PR branch ichinaski/zstd with baseline commit f97b42e58898aaeec219de03e52a456d09628c1d in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 17 '24 13:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-17 13:23:40

Comparing candidate commit a3eae6d7f38ecbc794b523af7eb5201c78b997b7 in PR branch ichinaski/zstd with baseline commit f97b42e58898aaeec219de03e52a456d09628c1d in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 17 '24 13:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-17 15:40:18

Comparing candidate commit 95f2eea9ef357eb0fd516930717098ce5928d05b in PR branch ichinaski/zstd with baseline commit 24d26177ad9d225e2e7103e895463b07573f7faf in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 17 '24 15:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-18 15:37:53

Comparing candidate commit ffb6876264f2013cbce7feb158bc760a59c1f9bb in PR branch ichinaski/zstd with baseline commit a975ff101886e6702a0e98def657b7503617dd2c in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 18 '24 15:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-19 10:12:18

Comparing candidate commit ffb6876264f2013cbce7feb158bc760a59c1f9bb in PR branch ichinaski/zstd with baseline commit 9cddecd4ab48283633f5091e47d468be546cdbe5 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 19 '24 10:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-19 10:30:35

Comparing candidate commit a1b8de3635672a754c599a0f9d3b090f3bc492d2 in PR branch ichinaski/zstd with baseline commit 9cddecd4ab48283633f5091e47d468be546cdbe5 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 19 '24 10:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-19 10:37:29

Comparing candidate commit 5e07af087c006e2cd5db63477ac93ab1d6c23d52 in PR branch ichinaski/zstd with baseline commit a1d61bc7e3b5caf22cdb9dcda3708140d399b688 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 19 '24 10:06 pr-commenter[bot]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=38293481 --os-family=ubuntu

Note: This applies to commit 6582978b

Jun 19 '24 11:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-20 18:06:26

Comparing candidate commit 7d57367a899880a315d47fa6a92fe90eff818399 in PR branch ichinaski/zstd with baseline commit 672608593f7154a5b7f6331199a179e01e1691ab in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 20 '24 18:06 pr-commenter[bot]

Regression Detector

Regression Detector Results

Run ID: 68ed8df1-00da-4681-9720-39436917a242 Metrics dashboard Target profiles

Baseline: 147c4943f564a0b808f3bed5fd0aaa6765b29f02 Comparison: 6582978b59f3d4a17703fabeaadeeb24514812c6

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	pycheck_1000_100byte_tags	% cpu utilization	+3.34	[-1.54, +8.22]	Logs
➖	otel_to_otel_logs	ingress throughput	+0.59	[-0.22, +1.40]	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.00	[-0.00, +0.00]	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	Logs
➖	idle	memory utilization	-0.00	[-0.05, +0.04]	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.30	[-1.20, +0.60]	Logs
➖	basic_py_check	% cpu utilization	-0.73	[-3.39, +1.94]	Logs
➖	file_tree	memory utilization	-1.61	[-1.73, -1.49]	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-2.35	[-14.85, +10.15]	Logs

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

Jun 20 '24 19:06 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-06-21 09:19:01

Comparing candidate commit 7d57367a899880a315d47fa6a92fe90eff818399 in PR branch ichinaski/zstd with baseline commit c0a3b74ae80331c5a629ee179676aeed5127add5 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jun 21 '24 09:06 pr-commenter[bot]

Go Package Import Differences

Baseline: 147c4943f564a0b808f3bed5fd0aaa6765b29f02 Comparison: 6582978b59f3d4a17703fabeaadeeb24514812c6

binary	os	arch	change
serverless	linux	amd64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd
serverless	linux	arm64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd
trace-agent	linux	amd64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd
trace-agent	linux	arm64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd
trace-agent	windows	amd64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd
trace-agent	darwin	amd64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd
trace-agent	darwin	arm64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd
heroku-trace-agent	linux	amd64	+3, -0 +github.com/DataDog/datadog-agent/comp/trace/compression/def +github.com/DataDog/datadog-agent/comp/trace/compression/impl-gzip +github.com/DataDog/datadog-agent/comp/trace/compression/impl-zstd

Jul 01 '24 10:07 cit-pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-07-01 10:36:25

Comparing candidate commit 42767df0cab669bd16c74450ab8c2a1681fd1629 in PR branch ichinaski/zstd with baseline commit 2274b1f5cd0a5a93a6931a84920c1d049502bc68 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jul 01 '24 10:07 pr-commenter[bot]

:warning::rotating_light: Warning, this pull request increases the binary size of serverless extension by 33216 bytes. Each MB of binary size increase means about 10ms of additional cold start time, so this pull request would increase cold start time by 0ms.

If you have questions, we are happy to help, come visit us in the #serverless slack channel and provide a link to this comment.

Debug info

These dependencies were added to the serverless extension by this pull request:

View dependency graphs for each added dependency in the artifacts section of the github action.

We suggest you consider adding the !serverless build tag to remove any new dependencies not needed in the serverless extension.

Jul 01 '24 11:07 github-actions[bot]

Benchmarks

Benchmark execution time: 2024-07-01 11:22:10

Comparing candidate commit 9e76fb1193c72bd5be8aa511c480698135e44877 in PR branch ichinaski/zstd with baseline commit d2054c27123125b38fe349616fb80b479233a1c3 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jul 01 '24 11:07 pr-commenter[bot]

:warning::rotating_light: Warning, this pull request increases the binary size of serverless extension by 33216 bytes. Each MB of binary size increase means about 10ms of additional cold start time, so this pull request would increase cold start time by 0ms.

If you have questions, we are happy to help, come visit us in the #serverless slack channel and provide a link to this comment.

Debug info

These dependencies were added to the serverless extension by this pull request:

View dependency graphs for each added dependency in the artifacts section of the github action.

We suggest you consider adding the !serverless build tag to remove any new dependencies not needed in the serverless extension.

Jul 01 '24 11:07 github-actions[bot]

Benchmarks

Benchmark execution time: 2024-07-01 11:29:52

Comparing candidate commit 3e08be6167239a1430b7f411dcc1d619b345b9b6 in PR branch ichinaski/zstd with baseline commit e4c9ff675c13e4cca8080adcd65cda8f72986649 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jul 01 '24 11:07 pr-commenter[bot]

:warning::rotating_light: Warning, this pull request increases the binary size of serverless extension by 33216 bytes. Each MB of binary size increase means about 10ms of additional cold start time, so this pull request would increase cold start time by 0ms.

If you have questions, we are happy to help, come visit us in the #serverless slack channel and provide a link to this comment.

Debug info

These dependencies were added to the serverless extension by this pull request:

View dependency graphs for each added dependency in the artifacts section of the github action.

We suggest you consider adding the !serverless build tag to remove any new dependencies not needed in the serverless extension.

Jul 01 '24 11:07 github-actions[bot]

Benchmarks

Benchmark execution time: 2024-07-01 12:02:48

Comparing candidate commit 52c16b05137785a3271efd6c3186638f5526ef66 in PR branch ichinaski/zstd with baseline commit 680baa2f65d42947b2c362f75d4d223c2cdd9ce2 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jul 01 '24 12:07 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-07-01 19:13:55

Comparing candidate commit 1942bc7bc7b0d0fe0da030272f4c40009c17494e in PR branch ichinaski/zstd with baseline commit 827a81b23dd401dac360c9f27ecfd7882d6b8e67 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jul 01 '24 19:07 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-07-02 10:18:30

Comparing candidate commit bced0435d0edb0b7c7190e6a68b6704232f70321 in PR branch ichinaski/zstd with baseline commit f350ef14a5ecfaf059e313b7d87460aa24460a81 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jul 02 '24 10:07 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-07-02 10:44:32

Comparing candidate commit 7f5c1e84584822ec829cd880e0faf271a0f47331 in PR branch ichinaski/zstd with baseline commit db37d28ddcdd8fedfd1266443cbe30c2b2cd856e in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 1 unstable metrics.

Jul 02 '24 10:07 pr-commenter[bot]

LGTM. Thanks for making all the changes. QQ: Do you think we should use zstd for OTel Agent ? OTel Agent isn't used by any customers and we are still in testing phase.

I see no reason not to use the same approach as the trace-agent for now: if zstd-encoding feature is available, we use zstd, otherwise and by default, gzip.

If you think this makes sense I can add it to the otel-agent commands.

Jul 02 '24 13:07 ichinaski

Lets merge this in. There is no hurry to make the changes. We can default to zstd in follow up PR

Jul 02 '24 13:07 dineshg13

Benchmarks

Benchmark execution time: 2024-07-03 12:55:11

Comparing candidate commit edce31643f347a81308b844ab624a398fbc3ce33 in PR branch ichinaski/zstd with baseline commit b0622f42c067b62643dd57be7cb2976959a45a67 in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 1 metrics, 1 unstable metrics.

scenario:BenchmarkAgentTraceProcessing-24

🟥 allocated_mem [+1.530MB; +1.698MB] or [+62.739%; +69.596%]

Jul 03 '24 12:07 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-07-03 15:08:09

Comparing candidate commit f5c199993fa533972651da23c00c5d18277995f2 in PR branch ichinaski/zstd with baseline commit 8842b600cb442826f130b3cd08e910ee1f2b2e8f in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 2 unstable metrics.

Jul 03 '24 15:07 pr-commenter[bot]

Benchmarks

Benchmark execution time: 2024-07-03 17:04:44

Comparing candidate commit 13cedd017abe8ee5e1978ab6ef0fc1b2e489028f in PR branch ichinaski/zstd with baseline commit e1b4c3bc51796b66ab4e3df019ca70aaf18c0e4b in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

Jul 03 '24 17:07 pr-commenter[bot]

Serverless Benchmark Results

BenchmarkStartEndInvocation comparison between e1b4c3bc51796b66ab4e3df019ca70aaf18c0e4b and 899ebf20e45e97d7a1cac1b7819e15caefdfc2a0.

tl;dr

Use these benchmarks as an insight tool during development.

Skim down the vs base column in each chart. If there is a ~, then there was no statistically significant change to the benchmark. Otherwise, ensure the estimated percent change is either negative or very small.
The last row of each chart is the geomean. Ensure this percentage is either negative or very small.

What is this benchmarking?

The BenchmarkStartEndInvocation compares the amount of time it takes to call the start-invocation and end-invocation endpoints. For universal instrumentation languages (Dotnet, Golang, Java, Ruby), this represents the majority of the duration overhead added by our tracing layer.

The benchmark is run using a large variety of lambda request payloads. In the charts below, there is one row for each event payload type.

How do I interpret these charts?

The charts below comes from benchstat. They represent the statistical change in duration (sec/op), memory overhead (B/op), and allocations (allocs/op).

The benchstat docs explain how to interpret these charts.

Before the comparison table, we see common file-level configuration. If there are benchmarks with different configuration (for example, from different packages), benchstat will print separate tables for each configuration.

The table then compares the two input files for each benchmark. It shows the median and 95% confidence interval summaries for each benchmark before and after the change, and an A/B comparison under "vs base". ... The p-value measures how likely it is that any differences were due to random chance (i.e., noise). The "~" means benchstat did not detect a statistically significant difference between the two inputs. ...

Note that "statistically significant" is not the same as "large": with enough low-noise data, even very small changes can be distinguished from noise and considered statistically significant. It is, of course, generally easier to distinguish large changes from noise.

Finally, the last row of the table shows the geometric mean of each column, giving an overall picture of how the benchmarks changed. Proportional changes in the geomean reflect proportional changes in the benchmarks. For example, given n benchmarks, if sec/op for one of them increases by a factor of 2, then the sec/op geomean will increase by a factor of ⁿ√2.

I need more help

First off, do not worry if the benchmarks are failing. They are not tests. The intention is for them to be a tool for you to use during development.

If you would like a hand interpreting the results come chat with us in #serverless-agent in the internal DataDog slack or in #serverless in the public DataDog slack. We're happy to help!

Benchmark stats

goos: linux
goarch: amd64
pkg: github.com/DataDog/datadog-agent/pkg/serverless/daemon
cpu: AMD EPYC 7763 64-Core Processor                
                                      │ baseline/benchmark.log │       current/benchmark.log        │
                                      │         sec/op         │   sec/op     vs base               │
api-gateway-appsec.json                            83.81µ ± 5%   85.55µ ± 2%       ~ (p=0.218 n=10)
api-gateway-kong-appsec.json                       65.13µ ± 1%   66.08µ ± 1%  +1.46% (p=0.019 n=10)
api-gateway-kong.json                              63.47µ ± 1%   64.57µ ± 1%  +1.73% (p=0.001 n=10)
api-gateway-non-proxy-async.json                   101.1µ ± 2%   104.1µ ± 2%  +3.00% (p=0.000 n=10)
api-gateway-non-proxy.json                         100.8µ ± 2%   104.5µ ± 2%  +3.58% (p=0.000 n=10)
api-gateway-websocket-connect.json                 67.16µ ± 1%   68.76µ ± 2%  +2.38% (p=0.000 n=10)
api-gateway-websocket-default.json                 59.84µ ± 1%   62.93µ ± 2%  +5.16% (p=0.000 n=10)
api-gateway-websocket-disconnect.json              60.34µ ± 1%   62.84µ ± 1%  +4.15% (p=0.000 n=10)
api-gateway.json                                   112.8µ ± 1%   115.3µ ± 1%  +2.21% (p=0.001 n=10)
application-load-balancer.json                     61.19µ ± 1%   63.88µ ± 1%  +4.39% (p=0.000 n=10)
cloudfront.json                                    46.33µ ± 2%   47.68µ ± 2%  +2.93% (p=0.001 n=10)
cloudwatch-events.json                             37.04µ ± 1%   38.26µ ± 2%  +3.29% (p=0.000 n=10)
cloudwatch-logs.json                               63.67µ ± 1%   65.48µ ± 3%  +2.84% (p=0.000 n=10)
custom.json                                        30.16µ ± 2%   30.71µ ± 2%  +1.83% (p=0.019 n=10)
dynamodb.json                                      92.91µ ± 1%   95.31µ ± 1%  +2.59% (p=0.000 n=10)
empty.json                                         28.68µ ± 2%   28.95µ ± 1%  +0.92% (p=0.043 n=10)
eventbridge-custom.json                            41.74µ ± 2%   42.55µ ± 2%       ~ (p=0.075 n=10)
http-api.json                                      71.19µ ± 1%   73.09µ ± 2%  +2.67% (p=0.000 n=10)
kinesis-batch.json                                 69.72µ ± 1%   72.21µ ± 2%  +3.58% (p=0.000 n=10)
kinesis.json                                       54.33µ ± 1%   54.30µ ± 1%       ~ (p=0.684 n=10)
s3.json                                            59.27µ ± 1%   60.50µ ± 1%  +2.08% (p=0.001 n=10)
sns-batch.json                                     89.27µ ± 2%   92.01µ ± 1%  +3.07% (p=0.000 n=10)
sns.json                                           64.62µ ± 1%   65.69µ ± 1%  +1.65% (p=0.029 n=10)
snssqs.json                                        110.7µ ± 2%   114.3µ ± 1%  +3.17% (p=0.000 n=10)
snssqs_no_dd_context.json                          97.40µ ± 1%   99.90µ ± 2%  +2.56% (p=0.003 n=10)
sqs-aws-header.json                                54.66µ ± 1%   56.17µ ± 1%  +2.76% (p=0.000 n=10)
sqs-batch.json                                     94.18µ ± 1%   96.98µ ± 1%  +2.97% (p=0.000 n=10)
sqs.json                                           68.60µ ± 1%   70.51µ ± 2%  +2.78% (p=0.001 n=10)
sqs_no_dd_context.json                             61.35µ ± 3%   63.97µ ± 2%  +4.26% (p=0.001 n=10)
geomean                                            65.49µ        67.25µ       +2.68%

                                      │ baseline/benchmark.log │        current/benchmark.log        │
                                      │          B/op          │     B/op      vs base               │
api-gateway-appsec.json                           37.26Ki ± 0%   37.26Ki ± 0%       ~ (p=0.670 n=10)
api-gateway-kong-appsec.json                      26.91Ki ± 0%   26.92Ki ± 0%       ~ (p=0.271 n=10)
api-gateway-kong.json                             24.41Ki ± 0%   24.42Ki ± 0%       ~ (p=0.127 n=10)
api-gateway-non-proxy-async.json                  48.00Ki ± 0%   48.02Ki ± 0%       ~ (p=0.254 n=10)
api-gateway-non-proxy.json                        47.22Ki ± 0%   47.24Ki ± 0%       ~ (p=0.343 n=10)
api-gateway-websocket-connect.json                25.44Ki ± 0%   25.44Ki ± 0%       ~ (p=0.193 n=10)
api-gateway-websocket-default.json                21.35Ki ± 0%   21.36Ki ± 0%  +0.06% (p=0.006 n=10)
api-gateway-websocket-disconnect.json             21.12Ki ± 0%   21.14Ki ± 0%  +0.09% (p=0.000 n=10)
api-gateway.json                                  49.54Ki ± 0%   49.54Ki ± 0%       ~ (p=0.343 n=10)
application-load-balancer.json                    22.31Ki ± 0%   22.32Ki ± 0%  +0.06% (p=0.000 n=10)
cloudfront.json                                   17.63Ki ± 0%   17.65Ki ± 0%  +0.08% (p=0.017 n=10)
cloudwatch-events.json                            11.67Ki ± 0%   11.69Ki ± 0%  +0.15% (p=0.000 n=10)
cloudwatch-logs.json                              53.35Ki ± 0%   53.37Ki ± 0%       ~ (p=0.085 n=10)
custom.json                                       9.709Ki ± 0%   9.711Ki ± 0%       ~ (p=0.725 n=10)
dynamodb.json                                     40.67Ki ± 0%   40.66Ki ± 0%       ~ (p=0.753 n=10)
empty.json                                        9.273Ki ± 0%   9.281Ki ± 0%       ~ (p=0.118 n=10)
eventbridge-custom.json                           13.40Ki ± 0%   13.42Ki ± 0%  +0.08% (p=0.017 n=10)
http-api.json                                     23.70Ki ± 0%   23.69Ki ± 0%       ~ (p=0.811 n=10)
kinesis-batch.json                                27.00Ki ± 0%   27.03Ki ± 0%       ~ (p=0.050 n=10)
kinesis.json                                      17.79Ki ± 0%   17.79Ki ± 0%       ~ (p=0.542 n=10)
s3.json                                           20.34Ki ± 0%   20.35Ki ± 0%       ~ (p=0.239 n=10)
sns-batch.json                                    38.63Ki ± 0%   38.67Ki ± 0%       ~ (p=0.172 n=10)
sns.json                                          23.97Ki ± 0%   24.01Ki ± 0%       ~ (p=0.072 n=10)
snssqs.json                                       50.74Ki ± 0%   50.80Ki ± 0%       ~ (p=0.138 n=10)
snssqs_no_dd_context.json                         44.79Ki ± 0%   44.84Ki ± 0%  +0.11% (p=0.000 n=10)
sqs-aws-header.json                               18.79Ki ± 0%   18.85Ki ± 0%  +0.31% (p=0.018 n=10)
sqs-batch.json                                    41.59Ki ± 0%   41.63Ki ± 0%       ~ (p=0.382 n=10)
sqs.json                                          25.48Ki ± 0%   25.58Ki ± 0%  +0.40% (p=0.009 n=10)
sqs_no_dd_context.json                            20.61Ki ± 1%   20.68Ki ± 1%       ~ (p=0.247 n=10)
geomean                                           25.68Ki        25.71Ki       +0.09%

                                      │ baseline/benchmark.log │        current/benchmark.log        │
                                      │       allocs/op        │ allocs/op   vs base                 │
api-gateway-appsec.json                             630.0 ± 0%   630.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-kong-appsec.json                        488.0 ± 0%   488.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-kong.json                               466.0 ± 0%   466.0 ± 0%       ~ (p=1.000 n=10) ¹
api-gateway-non-proxy-async.json                    725.0 ± 0%   726.0 ± 0%       ~ (p=0.370 n=10)
api-gateway-non-proxy.json                          716.0 ± 0%   716.0 ± 0%       ~ (p=1.000 n=10)
api-gateway-websocket-connect.json                  453.0 ± 0%   453.0 ± 0%       ~ (p=0.474 n=10)
api-gateway-websocket-default.json                  379.0 ± 0%   379.0 ± 0%       ~ (p=0.474 n=10)
api-gateway-websocket-disconnect.json               369.0 ± 0%   370.0 ± 0%  +0.27% (p=0.005 n=10)
api-gateway.json                                    791.0 ± 0%   791.0 ± 0%       ~ (p=0.582 n=10)
application-load-balancer.json                      352.0 ± 0%   352.0 ± 0%       ~ (p=1.000 n=10)
cloudfront.json                                     283.5 ± 0%   284.0 ± 0%  +0.18% (p=0.033 n=10)
cloudwatch-events.json                              220.0 ± 0%   220.0 ± 0%       ~ (p=1.000 n=10)
cloudwatch-logs.json                                215.0 ± 0%   215.5 ± 0%       ~ (p=0.141 n=10)
custom.json                                         168.0 ± 1%   168.0 ± 0%       ~ (p=1.000 n=10)
dynamodb.json                                       589.0 ± 0%   589.0 ± 0%       ~ (p=1.000 n=10)
empty.json                                          159.5 ± 0%   160.0 ± 0%       ~ (p=0.141 n=10)
eventbridge-custom.json                             254.0 ± 0%   254.0 ± 0%       ~ (p=0.582 n=10)
http-api.json                                       432.0 ± 0%   432.0 ± 0%       ~ (p=0.628 n=10)
kinesis-batch.json                                  390.0 ± 0%   391.0 ± 0%       ~ (p=0.179 n=10)
kinesis.json                                        285.0 ± 0%   285.0 ± 0%       ~ (p=0.720 n=10)
s3.json                                             358.0 ± 0%   358.0 ± 0%       ~ (p=0.232 n=10)
sns-batch.json                                      455.0 ± 0%   455.0 ± 0%       ~ (p=0.191 n=10)
sns.json                                            323.0 ± 0%   323.5 ± 0%  +0.15% (p=0.009 n=10)
snssqs.json                                         450.0 ± 0%   451.0 ± 0%       ~ (p=0.053 n=10)
snssqs_no_dd_context.json                           399.0 ± 0%   400.0 ± 0%  +0.25% (p=0.000 n=10)
sqs-aws-header.json                                 274.0 ± 0%   274.5 ± 0%  +0.18% (p=0.044 n=10)
sqs-batch.json                                      503.0 ± 0%   503.5 ± 0%       ~ (p=0.406 n=10)
sqs.json                                            350.0 ± 0%   351.0 ± 0%  +0.29% (p=0.005 n=10)
sqs_no_dd_context.json                              323.5 ± 1%   324.5 ± 0%       ~ (p=0.396 n=10)
geomean                                             376.5        376.9       +0.10%
¹ all samples are equal

Jul 03 '24 17:07 github-actions[bot]

Benchmarks

Benchmark execution time: 2024-07-03 17:13:37

Comparing candidate commit 6582978b59f3d4a17703fabeaadeeb24514812c6 in PR branch ichinaski/zstd with baseline commit e1b4c3bc51796b66ab4e3df019ca70aaf18c0e4b in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

Jul 03 '24 17:07 pr-commenter[bot]

/merge

Jul 04 '24 13:07 ichinaski

[APM] Use Zstd compression on Trace Payloads

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Test changes on VM

Benchmarks

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Explanation

Benchmarks

Go Package Import Differences

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Benchmarks

Benchmarks

scenario:BenchmarkAgentTraceProcessing-24

Benchmarks

Benchmarks

Serverless Benchmark Results

Benchmarks