vector icon indicating copy to clipboard operation
vector copied to clipboard

feat(aws_kinesis_firehose): Pass `access_key` to Firehose source metadata

Open dizlv opened this issue 2 years ago • 3 comments

Adress #14107

dizlv avatar Jan 11 '23 14:01 dizlv

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

:white_check_mark: spencergilbert
:white_check_mark: dizlv
:x: Dmitrijs Zubriks


Dmitrijs Zubriks seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

bits-bot avatar Jan 11 '23 14:01 bits-bot

Deploy Preview for vector-project ready!

Name Link
Latest commit 8ad27e32850a98d350ee31d94a1414df1d74105d
Latest deploy log https://app.netlify.com/sites/vector-project/deploys/63c6c38bab0297000877c302
Deploy Preview https://deploy-preview-15904--vector-project.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify[bot] avatar Jan 11 '23 14:01 netlify[bot]

Deploy Preview for vrl-playground canceled.

Name Link
Latest commit 8ad27e32850a98d350ee31d94a1414df1d74105d
Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/63c6c38a2b9b640008b82d83

netlify[bot] avatar Jan 11 '23 14:01 netlify[bot]

Regression Test Results

Run ID: 6600f1de-669c-453d-992c-ff99f91d3692
Baseline: ce003848558b7ee602336c7f2f0feda9e675e0ab
Comparison: dac7973b29e5ff61077c6916c1377a53202a6f66
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_humio_metrics 68.16KiB/CPU-s 1.13 100.00% 5.89MiB/CPU-s 268.58KiB/CPU-s 3.47KiB/CPU-s 0.0 0.044554 5.95MiB/CPU-s 229.44KiB/CPU-s 2.96KiB/CPU-s 0.0 0.037635 False False
http_to_http_acks 53.78KiB/CPU-s 1.0 70.06% 5.24MiB/CPU-s 2.73MiB/CPU-s 36.07KiB/CPU-s 0.0 0.521044 5.29MiB/CPU-s 2.82MiB/CPU-s 37.21KiB/CPU-s 0.0 0.53221 True False
http_to_http_json 123.54KiB/CPU-s 0.9 100.00% 13.47MiB/CPU-s 380.03KiB/CPU-s 4.9KiB/CPU-s 0.0 0.027544 13.59MiB/CPU-s 262.01KiB/CPU-s 3.38KiB/CPU-s 0.0 0.018822 False False
datadog_agent_remap_blackhole 187.44KiB/CPU-s 0.61 100.00% 29.86MiB/CPU-s 2.0MiB/CPU-s 26.45KiB/CPU-s 0.0 0.066958 30.05MiB/CPU-s 1.82MiB/CPU-s 24.08KiB/CPU-s 0.0 0.060646 False False
otlp_http_to_blackhole 9.07KiB/CPU-s 0.59 99.99% 1.51MiB/CPU-s 123.63KiB/CPU-s 1.6KiB/CPU-s 0.0 0.079977 1.52MiB/CPU-s 126.01KiB/CPU-s 1.63KiB/CPU-s 0.0 0.081041 False False
otlp_grpc_to_blackhole 3.11KiB/CPU-s 0.3 99.95% 1.03MiB/CPU-s 45.71KiB/CPU-s 604.13B/CPU-s 0.0 0.043364 1.03MiB/CPU-s 52.24KiB/CPU-s 690.26B/CPU-s 0.0 0.049413 False False
syslog_log2metric_splunk_hec_metrics 19.76KiB/CPU-s 0.21 100.00% 9.39MiB/CPU-s 201.24KiB/CPU-s 2.6KiB/CPU-s 0.0 0.020934 9.41MiB/CPU-s 299.01KiB/CPU-s 3.86KiB/CPU-s 0.0 0.03104 False False
enterprise_http_to_http 6.11KiB/CPU-s 0.04 78.49% 13.61MiB/CPU-s 306.69KiB/CPU-s 3.96KiB/CPU-s 0.0 0.021998 13.62MiB/CPU-s 227.94KiB/CPU-s 2.94KiB/CPU-s 0.0 0.016342 False False
syslog_humio_logs 3.33KiB/CPU-s 0.04 34.89% 8.99MiB/CPU-s 453.56KiB/CPU-s 5.85KiB/CPU-s 0.0 0.049239 9.0MiB/CPU-s 345.11KiB/CPU-s 4.46KiB/CPU-s 0.0 0.037453 False False
splunk_hec_to_splunk_hec_logs_noack 2.29KiB/CPU-s 0.02 47.11% 13.62MiB/CPU-s 206.56KiB/CPU-s 2.67KiB/CPU-s 0.0 0.014806 13.63MiB/CPU-s 190.9KiB/CPU-s 2.46KiB/CPU-s 0.0 0.013681 False False
file_to_blackhole 7.52KiB/CPU-s 0.01 28.04% 54.49MiB/CPU-s 1.09MiB/CPU-s 14.44KiB/CPU-s 0.0 0.020055 54.5MiB/CPU-s 1.15MiB/CPU-s 15.18KiB/CPU-s 0.0 0.0211 False False
fluent_elasticsearch -225.66B/CPU-s -0.0 31.51% 45.41MiB/CPU-s 30.03KiB/CPU-s 392.37B/CPU-s 0.0 0.000646 45.41MiB/CPU-s 30.15KiB/CPU-s 393.98B/CPU-s 0.0 0.000648 False False
splunk_hec_to_splunk_hec_logs_acks 218.99B/CPU-s 0.0 2.66% 13.62MiB/CPU-s 340.69KiB/CPU-s 4.4KiB/CPU-s 0.0 0.024432 13.62MiB/CPU-s 362.77KiB/CPU-s 4.68KiB/CPU-s 0.0 0.026015 False False
http_to_http_noack -2.01KiB/CPU-s -0.01 26.90% 13.61MiB/CPU-s 310.25KiB/CPU-s 4.0KiB/CPU-s 0.0 0.022254 13.61MiB/CPU-s 331.62KiB/CPU-s 4.28KiB/CPU-s 0.0 0.02379 False False
splunk_hec_indexer_ack_blackhole -3.03KiB/CPU-s -0.02 50.81% 13.62MiB/CPU-s 235.14KiB/CPU-s 3.03KiB/CPU-s 0.0 0.016859 13.62MiB/CPU-s 247.26KiB/CPU-s 3.19KiB/CPU-s 0.0 0.017731 False False
syslog_splunk_hec_logs -28.31KiB/CPU-s -0.31 100.00% 8.92MiB/CPU-s 226.8KiB/CPU-s 2.93KiB/CPU-s 0.0 0.024819 8.9MiB/CPU-s 372.14KiB/CPU-s 4.8KiB/CPU-s 0.0 0.040849 False False
syslog_regex_logs2metric_ddmetrics -24.85KiB/CPU-s -0.66 99.64% 3.65MiB/CPU-s 522.77KiB/CPU-s 6.75KiB/CPU-s 0.0 0.139732 3.63MiB/CPU-s 403.61KiB/CPU-s 5.21KiB/CPU-s 0.0 0.108602 True False
splunk_hec_route_s3 -97.23KiB/CPU-s -0.83 100.00% 11.41MiB/CPU-s 768.77KiB/CPU-s 9.92KiB/CPU-s 0.0 0.065783 11.32MiB/CPU-s 770.14KiB/CPU-s 9.94KiB/CPU-s 0.0 0.066452 False False
datadog_agent_remap_blackhole_acks -275.22KiB/CPU-s -0.86 100.00% 31.42MiB/CPU-s 807.0KiB/CPU-s 10.42KiB/CPU-s 0.0 0.025078 31.15MiB/CPU-s 902.21KiB/CPU-s 11.65KiB/CPU-s 0.0 0.028278 False False
socket_to_socket_blackhole -147.56KiB/CPU-s -1.09 100.00% 13.2MiB/CPU-s 365.59KiB/CPU-s 4.72KiB/CPU-s 0.0 0.027054 13.05MiB/CPU-s 299.14KiB/CPU-s 3.86KiB/CPU-s 0.0 0.022381 False False
http_text_to_http_json -476.87KiB/CPU-s -1.91 100.00% 24.42MiB/CPU-s 839.63KiB/CPU-s 10.84KiB/CPU-s 0.0 0.033579 23.95MiB/CPU-s 951.34KiB/CPU-s 12.28KiB/CPU-s 0.0 0.038787 False False
datadog_agent_remap_datadog_logs_acks -809.91KiB/CPU-s -2.31 100.00% 34.29MiB/CPU-s 1.09MiB/CPU-s 14.39KiB/CPU-s 0.0 0.031772 33.5MiB/CPU-s 1.13MiB/CPU-s 14.88KiB/CPU-s 0.0 0.033611 False False
datadog_agent_remap_datadog_logs -949.15KiB/CPU-s -2.72 100.00% 34.07MiB/CPU-s 1.35MiB/CPU-s 17.88KiB/CPU-s 0.0 0.039731 33.14MiB/CPU-s 1.64MiB/CPU-s 21.71KiB/CPU-s 0.0 0.049587 False False
syslog_loki -267.04KiB/CPU-s -3.0 100.00% 8.68MiB/CPU-s 296.4KiB/CPU-s 3.83KiB/CPU-s 0.0 0.033328 8.42MiB/CPU-s 405.95KiB/CPU-s 5.24KiB/CPU-s 0.0 0.047058 False False

github-actions[bot] avatar Jan 13 '23 00:01 github-actions[bot]

Regression Test Results

Run ID: b99e1612-1fe2-43ee-b8dd-b665bb1115b9
Baseline: ce003848558b7ee602336c7f2f0feda9e675e0ab
Comparison: 116844eaed95f0d0dd02b6bbac373c42da0cda8e
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_loki 267.34KiB/CPU-s 3.01 100.00% 8.67MiB/CPU-s 226.78KiB/CPU-s 2.93KiB/CPU-s 0.0 0.025535 8.93MiB/CPU-s 104.26KiB/CPU-s 1.35KiB/CPU-s 0.0 0.011396 False False
socket_to_socket_blackhole 152.26KiB/CPU-s 1.13 100.00% 13.16MiB/CPU-s 279.0KiB/CPU-s 3.6KiB/CPU-s 0.0 0.020709 13.3MiB/CPU-s 82.08KiB/CPU-s 1.06KiB/CPU-s 0.0 0.006024 False False
syslog_splunk_hec_logs 91.51KiB/CPU-s 0.98 100.00% 9.12MiB/CPU-s 91.71KiB/CPU-s 1.18KiB/CPU-s 0.0 0.009824 9.21MiB/CPU-s 207.58KiB/CPU-s 2.68KiB/CPU-s 0.0 0.022021 False False
syslog_humio_logs 88.27KiB/CPU-s 0.95 100.00% 9.04MiB/CPU-s 322.0KiB/CPU-s 4.16KiB/CPU-s 0.0 0.034774 9.13MiB/CPU-s 246.52KiB/CPU-s 3.18KiB/CPU-s 0.0 0.026371 False False
syslog_log2metric_humio_metrics 53.61KiB/CPU-s 0.86 100.00% 6.09MiB/CPU-s 331.9KiB/CPU-s 4.28KiB/CPU-s 0.0 0.053225 6.14MiB/CPU-s 275.5KiB/CPU-s 3.56KiB/CPU-s 0.0 0.043803 False False
syslog_log2metric_splunk_hec_metrics 56.71KiB/CPU-s 0.61 100.00% 9.11MiB/CPU-s 396.63KiB/CPU-s 5.12KiB/CPU-s 0.0 0.042531 9.16MiB/CPU-s 533.48KiB/CPU-s 6.89KiB/CPU-s 0.0 0.05686 False False
otlp_grpc_to_blackhole 4.89KiB/CPU-s 0.46 100.00% 1.03MiB/CPU-s 44.7KiB/CPU-s 590.96B/CPU-s 0.0 0.042352 1.04MiB/CPU-s 54.6KiB/CPU-s 721.42B/CPU-s 0.0 0.051489 False False
http_to_http_json 52.43KiB/CPU-s 0.38 100.00% 13.57MiB/CPU-s 293.5KiB/CPU-s 3.79KiB/CPU-s 0.0 0.021118 13.62MiB/CPU-s 211.41KiB/CPU-s 2.73KiB/CPU-s 0.0 0.015155 False False
splunk_hec_to_splunk_hec_logs_noack 8.51KiB/CPU-s 0.06 96.16% 13.62MiB/CPU-s 256.45KiB/CPU-s 3.31KiB/CPU-s 0.0 0.018391 13.62MiB/CPU-s 188.86KiB/CPU-s 2.44KiB/CPU-s 0.0 0.013536 False False
otlp_http_to_blackhole 1.01KiB/CPU-s 0.06 35.63% 1.53MiB/CPU-s 118.75KiB/CPU-s 1.53KiB/CPU-s 0.0 0.076014 1.53MiB/CPU-s 119.72KiB/CPU-s 1.54KiB/CPU-s 0.0 0.076583 False False
enterprise_http_to_http 1.98KiB/CPU-s 0.01 35.04% 13.62MiB/CPU-s 252.82KiB/CPU-s 3.26KiB/CPU-s 0.0 0.018128 13.62MiB/CPU-s 223.54KiB/CPU-s 2.89KiB/CPU-s 0.0 0.016026 False False
splunk_hec_indexer_ack_blackhole -655.74B/CPU-s -0.0 11.17% 13.62MiB/CPU-s 250.33KiB/CPU-s 3.23KiB/CPU-s 0.0 0.017951 13.62MiB/CPU-s 249.63KiB/CPU-s 3.22KiB/CPU-s 0.0 0.017901 False False
splunk_hec_to_splunk_hec_logs_acks 72.64B/CPU-s 0.0 0.86% 13.61MiB/CPU-s 353.59KiB/CPU-s 4.56KiB/CPU-s 0.0 0.025361 13.61MiB/CPU-s 366.49KiB/CPU-s 4.73KiB/CPU-s 0.0 0.026286 False False
fluent_elasticsearch 134.9B/CPU-s 0.0 19.11% 45.41MiB/CPU-s 30.21KiB/CPU-s 394.54B/CPU-s 0.0 0.00065 45.41MiB/CPU-s 30.19KiB/CPU-s 394.37B/CPU-s 0.0 0.000649 False False
file_to_blackhole -1.75KiB/CPU-s -0.0 6.28% 54.49MiB/CPU-s 1.14MiB/CPU-s 15.12KiB/CPU-s 0.0 0.021 54.49MiB/CPU-s 1.24MiB/CPU-s 16.32KiB/CPU-s 0.0 0.022698 False False
http_to_http_acks -1.89KiB/CPU-s -0.04 2.91% 5.24MiB/CPU-s 2.76MiB/CPU-s 36.54KiB/CPU-s 0.0 0.527714 5.24MiB/CPU-s 2.79MiB/CPU-s 36.83KiB/CPU-s 0.0 0.532079 True False
http_to_http_noack -6.37KiB/CPU-s -0.05 80.68% 13.62MiB/CPU-s 225.29KiB/CPU-s 2.91KiB/CPU-s 0.0 0.016152 13.61MiB/CPU-s 305.3KiB/CPU-s 3.94KiB/CPU-s 0.0 0.021899 False False
datadog_agent_remap_datadog_logs -243.7KiB/CPU-s -0.7 100.00% 34.15MiB/CPU-s 1.11MiB/CPU-s 14.71KiB/CPU-s 0.0 0.032592 33.92MiB/CPU-s 1.19MiB/CPU-s 15.7KiB/CPU-s 0.0 0.03505 False False
datadog_agent_remap_datadog_logs_acks -257.41KiB/CPU-s -0.75 100.00% 33.42MiB/CPU-s 1.71MiB/CPU-s 22.6KiB/CPU-s 0.0 0.051183 33.17MiB/CPU-s 1.68MiB/CPU-s 22.22KiB/CPU-s 0.0 0.05069 False False
splunk_hec_route_s3 -157.74KiB/CPU-s -1.32 100.00% 11.64MiB/CPU-s 645.64KiB/CPU-s 8.33KiB/CPU-s 0.0 0.054176 11.48MiB/CPU-s 699.3KiB/CPU-s 9.02KiB/CPU-s 0.0 0.059466 False False
datadog_agent_remap_blackhole_acks -888.06KiB/CPU-s -2.75 100.00% 31.56MiB/CPU-s 1.11MiB/CPU-s 14.68KiB/CPU-s 0.0 0.035159 30.7MiB/CPU-s 1.63MiB/CPU-s 21.56KiB/CPU-s 0.0 0.053135 False False
http_text_to_http_json -717.04KiB/CPU-s -2.88 100.00% 24.33MiB/CPU-s 917.04KiB/CPU-s 11.84KiB/CPU-s 0.0 0.036811 23.63MiB/CPU-s 954.22KiB/CPU-s 12.31KiB/CPU-s 0.0 0.039438 False False
syslog_regex_logs2metric_ddmetrics -147.23KiB/CPU-s -3.9 100.00% 3.68MiB/CPU-s 503.76KiB/CPU-s 6.5KiB/CPU-s 0.0 0.133598 3.54MiB/CPU-s 553.8KiB/CPU-s 7.15KiB/CPU-s 0.0 0.152836 True False
datadog_agent_remap_blackhole -1.55MiB/CPU-s -4.97 100.00% 31.2MiB/CPU-s 1.05MiB/CPU-s 13.87KiB/CPU-s 0.0 0.033612 29.65MiB/CPU-s 1.59MiB/CPU-s 21.06KiB/CPU-s 0.0 0.053742 False False

github-actions[bot] avatar Jan 13 '23 00:01 github-actions[bot]

Regression Test Results

Run ID: 0b5aa722-514e-48a4-9337-752496172af9
Baseline: ce003848558b7ee602336c7f2f0feda9e675e0ab
Comparison: ae9a19a002425bb03d90697db991b6bfe6c04f48
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole 669.94KiB/CPU-s 2.13 100.00% 30.73MiB/CPU-s 1.67MiB/CPU-s 22.05KiB/CPU-s 0.0 0.054288 31.39MiB/CPU-s 1.04MiB/CPU-s 13.7KiB/CPU-s 0.0 0.033024 False False
datadog_agent_remap_blackhole_acks 662.25KiB/CPU-s 2.09 100.00% 30.88MiB/CPU-s 1.47MiB/CPU-s 19.42KiB/CPU-s 0.0 0.047558 31.53MiB/CPU-s 1.17MiB/CPU-s 15.51KiB/CPU-s 0.0 0.037202 False False
syslog_loki 132.76KiB/CPU-s 1.5 100.00% 8.63MiB/CPU-s 218.52KiB/CPU-s 2.82KiB/CPU-s 0.0 0.02474 8.75MiB/CPU-s 222.03KiB/CPU-s 2.87KiB/CPU-s 0.0 0.024764 False False
syslog_humio_logs 129.18KiB/CPU-s 1.4 100.00% 8.98MiB/CPU-s 166.65KiB/CPU-s 2.15KiB/CPU-s 0.0 0.018112 9.11MiB/CPU-s 286.06KiB/CPU-s 3.69KiB/CPU-s 0.0 0.03066 False False
http_to_http_acks 71.39KiB/CPU-s 1.33 82.59% 5.24MiB/CPU-s 2.78MiB/CPU-s 36.78KiB/CPU-s 0.0 0.531106 5.31MiB/CPU-s 2.84MiB/CPU-s 37.51KiB/CPU-s 0.0 0.534533 True False
syslog_splunk_hec_logs 116.29KiB/CPU-s 1.28 100.00% 8.88MiB/CPU-s 197.06KiB/CPU-s 2.54KiB/CPU-s 0.0 0.021668 8.99MiB/CPU-s 351.82KiB/CPU-s 4.54KiB/CPU-s 0.0 0.038196 False False
http_to_http_json 53.48KiB/CPU-s 0.39 100.00% 13.52MiB/CPU-s 342.67KiB/CPU-s 4.42KiB/CPU-s 0.0 0.024741 13.58MiB/CPU-s 290.74KiB/CPU-s 3.75KiB/CPU-s 0.0 0.020911 False False
enterprise_http_to_http 14.54KiB/CPU-s 0.1 99.67% 13.61MiB/CPU-s 353.08KiB/CPU-s 4.56KiB/CPU-s 0.0 0.025333 13.62MiB/CPU-s 148.29KiB/CPU-s 1.91KiB/CPU-s 0.0 0.010629 False False
splunk_hec_to_splunk_hec_logs_noack 4.64KiB/CPU-s 0.03 77.23% 13.62MiB/CPU-s 228.45KiB/CPU-s 2.95KiB/CPU-s 0.0 0.016378 13.63MiB/CPU-s 191.38KiB/CPU-s 2.47KiB/CPU-s 0.0 0.013716 False False
splunk_hec_to_splunk_hec_logs_acks 733.26B/CPU-s 0.01 8.87% 13.62MiB/CPU-s 346.18KiB/CPU-s 4.47KiB/CPU-s 0.0 0.024827 13.62MiB/CPU-s 358.15KiB/CPU-s 4.62KiB/CPU-s 0.0 0.025684 False False
fluent_elasticsearch 307.61B/CPU-s 0.0 42.08% 45.41MiB/CPU-s 30.0KiB/CPU-s 392.12B/CPU-s 0.0 0.000645 45.41MiB/CPU-s 30.02KiB/CPU-s 392.29B/CPU-s 0.0 0.000645 False False
splunk_hec_indexer_ack_blackhole 175.32B/CPU-s 0.0 3.00% 13.62MiB/CPU-s 248.99KiB/CPU-s 3.21KiB/CPU-s 0.0 0.017855 13.62MiB/CPU-s 250.41KiB/CPU-s 3.23KiB/CPU-s 0.0 0.017957 False False
file_to_blackhole -6.53KiB/CPU-s -0.01 23.64% 54.49MiB/CPU-s 1.17MiB/CPU-s 15.42KiB/CPU-s 0.0 0.021427 54.49MiB/CPU-s 1.16MiB/CPU-s 15.28KiB/CPU-s 0.0 0.021237 False False
http_to_http_noack -3.87KiB/CPU-s -0.03 54.94% 13.62MiB/CPU-s 259.18KiB/CPU-s 3.35KiB/CPU-s 0.0 0.018585 13.61MiB/CPU-s 301.49KiB/CPU-s 3.89KiB/CPU-s 0.0 0.021625 False False
syslog_regex_logs2metric_ddmetrics -17.08KiB/CPU-s -0.47 95.44% 3.59MiB/CPU-s 472.45KiB/CPU-s 6.1KiB/CPU-s 0.0 0.128655 3.57MiB/CPU-s 463.14KiB/CPU-s 5.98KiB/CPU-s 0.0 0.126708 True False
http_text_to_http_json -210.56KiB/CPU-s -0.84 100.00% 24.41MiB/CPU-s 743.0KiB/CPU-s 9.59KiB/CPU-s 0.0 0.029717 24.21MiB/CPU-s 589.73KiB/CPU-s 7.61KiB/CPU-s 0.0 0.023787 False False
otlp_http_to_blackhole -15.47KiB/CPU-s -0.99 100.00% 1.53MiB/CPU-s 114.7KiB/CPU-s 1.48KiB/CPU-s 0.0 0.073204 1.51MiB/CPU-s 128.83KiB/CPU-s 1.66KiB/CPU-s 0.0 0.083042 False False
otlp_grpc_to_blackhole -13.19KiB/CPU-s -1.25 100.00% 1.03MiB/CPU-s 47.46KiB/CPU-s 627.37B/CPU-s 0.0 0.044963 1.02MiB/CPU-s 57.9KiB/CPU-s 764.93B/CPU-s 0.0 0.055538 False False
splunk_hec_route_s3 -160.6KiB/CPU-s -1.34 100.00% 11.7MiB/CPU-s 573.51KiB/CPU-s 7.4KiB/CPU-s 0.0 0.047873 11.54MiB/CPU-s 783.35KiB/CPU-s 10.11KiB/CPU-s 0.0 0.066278 False False
syslog_log2metric_splunk_hec_metrics -128.02KiB/CPU-s -1.35 100.00% 9.25MiB/CPU-s 405.99KiB/CPU-s 5.24KiB/CPU-s 0.0 0.04287 9.12MiB/CPU-s 595.43KiB/CPU-s 7.69KiB/CPU-s 0.0 0.063735 False False
datadog_agent_remap_datadog_logs -685.79KiB/CPU-s -1.98 100.00% 33.79MiB/CPU-s 1.9MiB/CPU-s 25.14KiB/CPU-s 0.0 0.056323 33.12MiB/CPU-s 1.56MiB/CPU-s 20.59KiB/CPU-s 0.0 0.047059 False False
datadog_agent_remap_datadog_logs_acks -899.75KiB/CPU-s -2.6 100.00% 33.84MiB/CPU-s 1.47MiB/CPU-s 19.46KiB/CPU-s 0.0 0.043529 32.96MiB/CPU-s 1.53MiB/CPU-s 20.26KiB/CPU-s 0.0 0.04651 False False
socket_to_socket_blackhole -397.32KiB/CPU-s -2.91 100.00% 13.32MiB/CPU-s 213.76KiB/CPU-s 2.76KiB/CPU-s 0.0 0.015671 12.93MiB/CPU-s 439.06KiB/CPU-s 5.67KiB/CPU-s 0.0 0.033154 False False
syslog_log2metric_humio_metrics -266.37KiB/CPU-s -4.24 100.00% 6.14MiB/CPU-s 201.33KiB/CPU-s 2.6KiB/CPU-s 0.0 0.032025 5.88MiB/CPU-s 400.85KiB/CPU-s 5.18KiB/CPU-s 0.0 0.066584 False False

github-actions[bot] avatar Jan 13 '23 00:01 github-actions[bot]

Regression Test Results

Run ID: d5913a52-d354-45dd-a329-8eda672314b2
Baseline: 6d05e13337bf8daacf47f0e289a521408fe079cc
Comparison: b7516b03bdbc3b822b82f93ac11409d3f0e21ffb
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
syslog_log2metric_splunk_hec_metrics 930.59KiB/CPU-s 10.52 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_splunk_hec_metrics 930.59KiB/CPU-s 10.52 100.00% 8.64MiB/CPU-s 2.56MiB/CPU-s 33.86KiB/CPU-s 0.0 0.296555 9.55MiB/CPU-s 208.7KiB/CPU-s 2.69KiB/CPU-s 0.0 0.021343 True False
syslog_humio_logs 165.23KiB/CPU-s 1.8 100.00% 8.97MiB/CPU-s 267.1KiB/CPU-s 3.45KiB/CPU-s 0.0 0.029085 9.13MiB/CPU-s 273.64KiB/CPU-s 3.53KiB/CPU-s 0.0 0.02927 False False
splunk_hec_route_s3 188.39KiB/CPU-s 1.61 100.00% 11.44MiB/CPU-s 551.03KiB/CPU-s 7.11KiB/CPU-s 0.0 0.047049 11.62MiB/CPU-s 566.76KiB/CPU-s 7.32KiB/CPU-s 0.0 0.047626 False False
http_to_http_json 117.29KiB/CPU-s 0.85 100.00% 13.51MiB/CPU-s 393.28KiB/CPU-s 5.07KiB/CPU-s 0.0 0.028431 13.62MiB/CPU-s 210.78KiB/CPU-s 2.72KiB/CPU-s 0.0 0.01511 False False
http_to_http_acks 29.88KiB/CPU-s 0.56 44.21% 5.21MiB/CPU-s 2.7MiB/CPU-s 35.7KiB/CPU-s 0.0 0.518061 5.24MiB/CPU-s 2.76MiB/CPU-s 36.42KiB/CPU-s 0.0 0.525363 True False
datadog_agent_remap_blackhole_acks 67.6KiB/CPU-s 0.21 98.48% 31.17MiB/CPU-s 1.62MiB/CPU-s 21.47KiB/CPU-s 0.0 0.052089 31.24MiB/CPU-s 1.34MiB/CPU-s 17.74KiB/CPU-s 0.0 0.042961 False False
enterprise_http_to_http 10.17KiB/CPU-s 0.07 91.79% 13.61MiB/CPU-s 372.16KiB/CPU-s 4.8KiB/CPU-s 0.0 0.026706 13.62MiB/CPU-s 258.73KiB/CPU-s 3.34KiB/CPU-s 0.0 0.018553 False False
splunk_hec_to_splunk_hec_logs_noack 5.63KiB/CPU-s 0.04 85.38% 13.62MiB/CPU-s 229.67KiB/CPU-s 2.96KiB/CPU-s 0.0 0.016467 13.63MiB/CPU-s 193.25KiB/CPU-s 2.49KiB/CPU-s 0.0 0.01385 False False
splunk_hec_indexer_ack_blackhole 949.08B/CPU-s 0.01 16.28% 13.62MiB/CPU-s 249.24KiB/CPU-s 3.22KiB/CPU-s 0.0 0.017873 13.62MiB/CPU-s 245.16KiB/CPU-s 3.16KiB/CPU-s 0.0 0.017579 False False
file_to_blackhole 7.98KiB/CPU-s 0.01 29.99% 54.5MiB/CPU-s 1.08MiB/CPU-s 14.24KiB/CPU-s 0.0 0.019778 54.5MiB/CPU-s 1.14MiB/CPU-s 15.07KiB/CPU-s 0.0 0.020942 False False
splunk_hec_to_splunk_hec_logs_acks 989.96B/CPU-s 0.01 11.37% 13.61MiB/CPU-s 371.24KiB/CPU-s 4.79KiB/CPU-s 0.0 0.026626 13.62MiB/CPU-s 370.21KiB/CPU-s 4.78KiB/CPU-s 0.0 0.02655 False False
fluent_elasticsearch 60.62B/CPU-s 0.0 8.68% 45.41MiB/CPU-s 30.55KiB/CPU-s 399.3B/CPU-s 0.0 0.000657 45.41MiB/CPU-s 29.6KiB/CPU-s 386.97B/CPU-s 0.0 0.000636 False False
http_to_http_noack -4.1KiB/CPU-s -0.03 50.02% 13.61MiB/CPU-s 310.88KiB/CPU-s 4.01KiB/CPU-s 0.0 0.022299 13.61MiB/CPU-s 353.67KiB/CPU-s 4.56KiB/CPU-s 0.0 0.025375 False False
syslog_splunk_hec_logs -8.09KiB/CPU-s -0.09 95.55% 9.0MiB/CPU-s 133.79KiB/CPU-s 1.73KiB/CPU-s 0.0 0.014524 8.99MiB/CPU-s 281.62KiB/CPU-s 3.63KiB/CPU-s 0.0 0.030598 False False
datadog_agent_remap_datadog_logs -85.73KiB/CPU-s -0.25 99.99% 34.14MiB/CPU-s 1.22MiB/CPU-s 16.12KiB/CPU-s 0.0 0.035736 34.06MiB/CPU-s 1.16MiB/CPU-s 15.34KiB/CPU-s 0.0 0.034083 False False
otlp_grpc_to_blackhole -8.55KiB/CPU-s -0.8 100.00% 1.04MiB/CPU-s 37.46KiB/CPU-s 495.23B/CPU-s 0.0 0.035103 1.03MiB/CPU-s 52.34KiB/CPU-s 691.62B/CPU-s 0.0 0.049437 False False
syslog_loki -94.8KiB/CPU-s -1.06 100.00% 8.74MiB/CPU-s 314.9KiB/CPU-s 4.06KiB/CPU-s 0.0 0.035184 8.65MiB/CPU-s 274.08KiB/CPU-s 3.54KiB/CPU-s 0.0 0.03095 False False
http_text_to_http_json -335.89KiB/CPU-s -1.33 100.00% 24.73MiB/CPU-s 586.97KiB/CPU-s 7.58KiB/CPU-s 0.0 0.023181 24.4MiB/CPU-s 579.22KiB/CPU-s 7.47KiB/CPU-s 0.0 0.023182 False False
syslog_regex_logs2metric_ddmetrics -53.8KiB/CPU-s -1.39 100.00% 3.77MiB/CPU-s 448.22KiB/CPU-s 5.79KiB/CPU-s 0.0 0.116049 3.72MiB/CPU-s 421.76KiB/CPU-s 5.44KiB/CPU-s 0.0 0.110741 True False
socket_to_socket_blackhole -255.07KiB/CPU-s -1.85 100.00% 13.48MiB/CPU-s 175.98KiB/CPU-s 2.27KiB/CPU-s 0.0 0.012744 13.24MiB/CPU-s 137.82KiB/CPU-s 1.78KiB/CPU-s 0.0 0.010168 False False
datadog_agent_remap_blackhole -849.01KiB/CPU-s -2.67 100.00% 31.0MiB/CPU-s 981.62KiB/CPU-s 12.67KiB/CPU-s 0.0 0.030923 30.17MiB/CPU-s 1.91MiB/CPU-s 25.2KiB/CPU-s 0.0 0.063205 False False
otlp_http_to_blackhole -44.63KiB/CPU-s -2.81 100.00% 1.55MiB/CPU-s 108.34KiB/CPU-s 1.4KiB/CPU-s 0.0 0.068173 1.51MiB/CPU-s 132.32KiB/CPU-s 1.71KiB/CPU-s 0.0 0.085665 False False
datadog_agent_remap_datadog_logs_acks -1.31MiB/CPU-s -3.87 100.00% 33.84MiB/CPU-s 1.22MiB/CPU-s 16.15KiB/CPU-s 0.0 0.036125 32.53MiB/CPU-s 1.78MiB/CPU-s 23.51KiB/CPU-s 0.0 0.054702 False False
syslog_log2metric_humio_metrics -274.09KiB/CPU-s -4.36 100.00% 6.14MiB/CPU-s 260.52KiB/CPU-s 3.36KiB/CPU-s 0.0 0.041425 5.87MiB/CPU-s 361.39KiB/CPU-s 4.67KiB/CPU-s 0.0 0.060081 False False

github-actions[bot] avatar Jan 13 '23 22:01 github-actions[bot]

Regression Test Results

Run ID: 8ff95cac-2d69-4075-b8ef-173bc3ce82d2
Baseline: 6d05e13337bf8daacf47f0e289a521408fe079cc
Comparison: 0a2e99972ebae09b3835ab817f6e6752a94c8e7e
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
datadog_agent_remap_blackhole 1.6MiB/CPU-s 5.4 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole 1.6MiB/CPU-s 5.4 100.00% 29.64MiB/CPU-s 1.99MiB/CPU-s 26.37KiB/CPU-s 0.0 0.067287 31.24MiB/CPU-s 504.8KiB/CPU-s 6.52KiB/CPU-s 0.0 0.015778 False False
syslog_humio_logs 373.86KiB/CPU-s 4.1 100.00% 8.9MiB/CPU-s 349.09KiB/CPU-s 4.51KiB/CPU-s 0.0 0.038302 9.26MiB/CPU-s 190.0KiB/CPU-s 2.45KiB/CPU-s 0.0 0.020025 False False
syslog_log2metric_humio_metrics 237.02KiB/CPU-s 3.82 100.00% 6.05MiB/CPU-s 416.01KiB/CPU-s 5.37KiB/CPU-s 0.0 0.067105 6.28MiB/CPU-s 96.72KiB/CPU-s 1.25KiB/CPU-s 0.0 0.015028 False False
syslog_loki 198.2KiB/CPU-s 2.24 100.00% 8.64MiB/CPU-s 381.14KiB/CPU-s 4.92KiB/CPU-s 0.0 0.043051 8.84MiB/CPU-s 179.59KiB/CPU-s 2.32KiB/CPU-s 0.0 0.019841 False False
splunk_hec_route_s3 227.41KiB/CPU-s 1.94 100.00% 11.46MiB/CPU-s 667.01KiB/CPU-s 8.6KiB/CPU-s 0.0 0.056823 11.68MiB/CPU-s 569.42KiB/CPU-s 7.35KiB/CPU-s 0.0 0.047587 False False
datadog_agent_remap_blackhole_acks 566.52KiB/CPU-s 1.76 100.00% 31.51MiB/CPU-s 687.95KiB/CPU-s 8.88KiB/CPU-s 0.0 0.02132 32.06MiB/CPU-s 893.37KiB/CPU-s 11.53KiB/CPU-s 0.0 0.027209 False False
syslog_regex_logs2metric_ddmetrics 48.53KiB/CPU-s 1.32 100.00% 3.58MiB/CPU-s 514.67KiB/CPU-s 6.64KiB/CPU-s 0.0 0.140402 3.63MiB/CPU-s 452.29KiB/CPU-s 5.84KiB/CPU-s 0.0 0.121771 True False
syslog_splunk_hec_logs 105.93KiB/CPU-s 1.16 100.00% 8.95MiB/CPU-s 279.37KiB/CPU-s 3.61KiB/CPU-s 0.0 0.030496 9.05MiB/CPU-s 303.38KiB/CPU-s 3.91KiB/CPU-s 0.0 0.032738 False False
syslog_log2metric_splunk_hec_metrics 53.04KiB/CPU-s 0.55 100.00% 9.38MiB/CPU-s 305.66KiB/CPU-s 3.95KiB/CPU-s 0.0 0.03182 9.43MiB/CPU-s 328.34KiB/CPU-s 4.24KiB/CPU-s 0.0 0.033993 False False
http_to_http_json 45.23KiB/CPU-s 0.33 100.00% 13.56MiB/CPU-s 325.5KiB/CPU-s 4.2KiB/CPU-s 0.0 0.023444 13.6MiB/CPU-s 249.46KiB/CPU-s 3.22KiB/CPU-s 0.0 0.017909 False False
http_to_http_acks 10.86KiB/CPU-s 0.2 16.28% 5.23MiB/CPU-s 2.81MiB/CPU-s 37.14KiB/CPU-s 0.0 0.537127 5.24MiB/CPU-s 2.85MiB/CPU-s 37.62KiB/CPU-s 0.0 0.542915 True False
splunk_hec_to_splunk_hec_logs_noack 4.35KiB/CPU-s 0.03 74.14% 13.62MiB/CPU-s 227.9KiB/CPU-s 2.94KiB/CPU-s 0.0 0.016338 13.63MiB/CPU-s 191.98KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013759 False False
http_to_http_noack 2.01KiB/CPU-s 0.01 31.79% 13.62MiB/CPU-s 283.42KiB/CPU-s 3.66KiB/CPU-s 0.0 0.020326 13.62MiB/CPU-s 254.78KiB/CPU-s 3.29KiB/CPU-s 0.0 0.018269 False False
splunk_hec_indexer_ack_blackhole 1.08KiB/CPU-s 0.01 18.31% 13.62MiB/CPU-s 257.48KiB/CPU-s 3.32KiB/CPU-s 0.0 0.018466 13.62MiB/CPU-s 255.13KiB/CPU-s 3.29KiB/CPU-s 0.0 0.018296 False False
splunk_hec_to_splunk_hec_logs_acks 823.37B/CPU-s 0.01 9.72% 13.61MiB/CPU-s 361.17KiB/CPU-s 4.66KiB/CPU-s 0.0 0.025905 13.61MiB/CPU-s 360.79KiB/CPU-s 4.65KiB/CPU-s 0.0 0.025876 False False
fluent_elasticsearch -163.8B/CPU-s -0.0 23.18% 45.41MiB/CPU-s 30.23KiB/CPU-s 395.15B/CPU-s 0.0 0.00065 45.41MiB/CPU-s 29.89KiB/CPU-s 390.82B/CPU-s 0.0 0.000643 False False
enterprise_http_to_http -1.89KiB/CPU-s -0.01 33.57% 13.62MiB/CPU-s 225.28KiB/CPU-s 2.91KiB/CPU-s 0.0 0.016151 13.62MiB/CPU-s 251.38KiB/CPU-s 3.24KiB/CPU-s 0.0 0.018025 False False
file_to_blackhole -17.24KiB/CPU-s -0.03 55.30% 54.49MiB/CPU-s 1.08MiB/CPU-s 14.2KiB/CPU-s 0.0 0.01973 54.48MiB/CPU-s 1.34MiB/CPU-s 17.67KiB/CPU-s 0.0 0.024581 False False
datadog_agent_remap_datadog_logs -74.23KiB/CPU-s -0.21 99.10% 33.87MiB/CPU-s 1.65MiB/CPU-s 21.84KiB/CPU-s 0.0 0.048824 33.8MiB/CPU-s 1.38MiB/CPU-s 18.18KiB/CPU-s 0.0 0.040713 False False
datadog_agent_remap_datadog_logs_acks -262.96KiB/CPU-s -0.76 100.00% 33.59MiB/CPU-s 1.13MiB/CPU-s 14.98KiB/CPU-s 0.0 0.033743 33.34MiB/CPU-s 1.14MiB/CPU-s 15.1KiB/CPU-s 0.0 0.034267 False False
http_text_to_http_json -256.36KiB/CPU-s -1.03 100.00% 24.25MiB/CPU-s 770.24KiB/CPU-s 9.94KiB/CPU-s 0.0 0.031014 24.0MiB/CPU-s 628.9KiB/CPU-s 8.11KiB/CPU-s 0.0 0.025587 False False
otlp_grpc_to_blackhole -11.81KiB/CPU-s -1.12 100.00% 1.03MiB/CPU-s 46.81KiB/CPU-s 618.75B/CPU-s 0.0 0.044439 1.02MiB/CPU-s 61.57KiB/CPU-s 813.48B/CPU-s 0.0 0.059111 False False
otlp_http_to_blackhole -29.9KiB/CPU-s -1.91 100.00% 1.53MiB/CPU-s 109.12KiB/CPU-s 1.41KiB/CPU-s 0.0 0.069567 1.5MiB/CPU-s 128.16KiB/CPU-s 1.65KiB/CPU-s 0.0 0.083293 False False
socket_to_socket_blackhole -301.65KiB/CPU-s -2.2 100.00% 13.42MiB/CPU-s 259.9KiB/CPU-s 3.35KiB/CPU-s 0.0 0.018916 13.12MiB/CPU-s 256.13KiB/CPU-s 3.31KiB/CPU-s 0.0 0.01906 False False

github-actions[bot] avatar Jan 13 '23 22:01 github-actions[bot]

Regression Test Results

Run ID: 810177ea-21be-402c-8df9-76ed5b07d913
Baseline: 6d05e13337bf8daacf47f0e289a521408fe079cc
Comparison: 061264378cd545fcfb5eda07373ed6adedc129f7
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole 1020.21KiB/CPU-s 3.27 100.00% 30.51MiB/CPU-s 1.29MiB/CPU-s 17.06KiB/CPU-s 0.0 0.042251 31.51MiB/CPU-s 822.76KiB/CPU-s 10.62KiB/CPU-s 0.0 0.025499 False False
syslog_splunk_hec_logs 198.59KiB/CPU-s 2.2 100.00% 8.82MiB/CPU-s 267.4KiB/CPU-s 3.45KiB/CPU-s 0.0 0.029614 9.01MiB/CPU-s 212.07KiB/CPU-s 2.74KiB/CPU-s 0.0 0.022981 False False
splunk_hec_route_s3 176.07KiB/CPU-s 1.48 100.00% 11.64MiB/CPU-s 643.54KiB/CPU-s 8.3KiB/CPU-s 0.0 0.05399 11.81MiB/CPU-s 522.8KiB/CPU-s 6.75KiB/CPU-s 0.0 0.043222 False False
syslog_log2metric_humio_metrics 45.25KiB/CPU-s 0.72 100.00% 6.1MiB/CPU-s 257.99KiB/CPU-s 3.33KiB/CPU-s 0.0 0.04131 6.14MiB/CPU-s 326.01KiB/CPU-s 4.21KiB/CPU-s 0.0 0.051826 False False
syslog_loki 23.61KiB/CPU-s 0.26 100.00% 8.76MiB/CPU-s 279.56KiB/CPU-s 3.61KiB/CPU-s 0.0 0.031145 8.79MiB/CPU-s 297.38KiB/CPU-s 3.84KiB/CPU-s 0.0 0.033044 False False
splunk_hec_to_splunk_hec_logs_noack 6.34KiB/CPU-s 0.05 89.10% 13.62MiB/CPU-s 239.92KiB/CPU-s 3.1KiB/CPU-s 0.0 0.017202 13.62MiB/CPU-s 190.7KiB/CPU-s 2.46KiB/CPU-s 0.0 0.013667 False False
file_to_blackhole 3.86KiB/CPU-s 0.01 15.40% 54.49MiB/CPU-s 1.07MiB/CPU-s 14.14KiB/CPU-s 0.0 0.019647 54.5MiB/CPU-s 1.06MiB/CPU-s 13.95KiB/CPU-s 0.0 0.019381 False False
fluent_elasticsearch -33.52B/CPU-s -0.0 4.78% 45.41MiB/CPU-s 30.51KiB/CPU-s 398.58B/CPU-s 0.0 0.000656 45.41MiB/CPU-s 30.0KiB/CPU-s 392.04B/CPU-s 0.0 0.000645 False False
http_to_http_noack 352.12B/CPU-s 0.0 5.45% 13.62MiB/CPU-s 280.08KiB/CPU-s 3.61KiB/CPU-s 0.0 0.020086 13.62MiB/CPU-s 270.57KiB/CPU-s 3.49KiB/CPU-s 0.0 0.019404 False False
splunk_hec_indexer_ack_blackhole 299.68B/CPU-s 0.0 5.12% 13.62MiB/CPU-s 250.07KiB/CPU-s 3.23KiB/CPU-s 0.0 0.017933 13.62MiB/CPU-s 249.3KiB/CPU-s 3.22KiB/CPU-s 0.0 0.017877 False False
enterprise_http_to_http -109.65B/CPU-s -0.0 2.45% 13.62MiB/CPU-s 191.91KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013757 13.62MiB/CPU-s 189.44KiB/CPU-s 2.45KiB/CPU-s 0.0 0.01358 False False
splunk_hec_to_splunk_hec_logs_acks -1.45KiB/CPU-s -0.01 17.86% 13.62MiB/CPU-s 351.39KiB/CPU-s 4.53KiB/CPU-s 0.0 0.025198 13.62MiB/CPU-s 350.12KiB/CPU-s 4.52KiB/CPU-s 0.0 0.025109 False False
http_to_http_json -7.58KiB/CPU-s -0.05 85.34% 13.58MiB/CPU-s 281.5KiB/CPU-s 3.63KiB/CPU-s 0.0 0.020247 13.57MiB/CPU-s 290.49KiB/CPU-s 3.75KiB/CPU-s 0.0 0.020905 False False
syslog_log2metric_splunk_hec_metrics -12.27KiB/CPU-s -0.13 92.03% 9.38MiB/CPU-s 326.29KiB/CPU-s 4.21KiB/CPU-s 0.0 0.033981 9.36MiB/CPU-s 433.21KiB/CPU-s 5.59KiB/CPU-s 0.0 0.045174 False False
syslog_regex_logs2metric_ddmetrics -9.8KiB/CPU-s -0.26 79.00% 3.68MiB/CPU-s 434.82KiB/CPU-s 5.61KiB/CPU-s 0.0 0.115381 3.67MiB/CPU-s 421.19KiB/CPU-s 5.44KiB/CPU-s 0.0 0.112055 True False
datadog_agent_remap_blackhole_acks -83.61KiB/CPU-s -0.26 99.99% 30.99MiB/CPU-s 625.6KiB/CPU-s 8.08KiB/CPU-s 0.0 0.019713 30.91MiB/CPU-s 1.48MiB/CPU-s 19.62KiB/CPU-s 0.0 0.048005 False False
syslog_humio_logs -58.99KiB/CPU-s -0.63 100.00% 9.15MiB/CPU-s 161.21KiB/CPU-s 2.08KiB/CPU-s 0.0 0.017204 9.09MiB/CPU-s 311.56KiB/CPU-s 4.02KiB/CPU-s 0.0 0.033458 False False
datadog_agent_remap_datadog_logs_acks -335.53KiB/CPU-s -0.96 100.00% 33.96MiB/CPU-s 1.12MiB/CPU-s 14.75KiB/CPU-s 0.0 0.032869 33.63MiB/CPU-s 1.06MiB/CPU-s 13.95KiB/CPU-s 0.0 0.031378 False False
http_to_http_acks -65.97KiB/CPU-s -1.22 79.46% 5.28MiB/CPU-s 2.76MiB/CPU-s 36.54KiB/CPU-s 0.0 0.523987 5.21MiB/CPU-s 2.81MiB/CPU-s 37.14KiB/CPU-s 0.0 0.539295 True False
datadog_agent_remap_datadog_logs -522.12KiB/CPU-s -1.5 100.00% 34.03MiB/CPU-s 1.46MiB/CPU-s 19.34KiB/CPU-s 0.0 0.043011 33.52MiB/CPU-s 1.87MiB/CPU-s 24.64KiB/CPU-s 0.0 0.055652 False False
otlp_http_to_blackhole -29.95KiB/CPU-s -1.9 100.00% 1.54MiB/CPU-s 107.97KiB/CPU-s 1.39KiB/CPU-s 0.0 0.068472 1.51MiB/CPU-s 121.91KiB/CPU-s 1.57KiB/CPU-s 0.0 0.078814 False False
otlp_grpc_to_blackhole -21.51KiB/CPU-s -2.01 100.00% 1.04MiB/CPU-s 37.84KiB/CPU-s 500.3B/CPU-s 0.0 0.035443 1.02MiB/CPU-s 59.14KiB/CPU-s 781.37B/CPU-s 0.0 0.056525 False False
socket_to_socket_blackhole -307.28KiB/CPU-s -2.26 100.00% 13.28MiB/CPU-s 313.91KiB/CPU-s 4.05KiB/CPU-s 0.0 0.023089 12.98MiB/CPU-s 344.45KiB/CPU-s 4.45KiB/CPU-s 0.0 0.025921 False False
http_text_to_http_json -586.62KiB/CPU-s -2.3 100.00% 24.9MiB/CPU-s 545.94KiB/CPU-s 7.05KiB/CPU-s 0.0 0.021411 24.33MiB/CPU-s 557.68KiB/CPU-s 7.2KiB/CPU-s 0.0 0.022387 False False

github-actions[bot] avatar Jan 13 '23 22:01 github-actions[bot]

Hey @jszwedko, is this going to land in 0.27? 👀

dizlv avatar Jan 17 '23 13:01 dizlv

Hey @jszwedko, is this going to land in 0.27? 👀

Hey! Unfortunately I don't think it'll make the cut for 0.27.0, which we've already cut but not released, but we release every 6 weeks so it won't be too long before it'll be included in 0.28.0. Thanks again for this contribution!

jszwedko avatar Jan 17 '23 14:01 jszwedko

Regression Test Results

Run ID: 8aad0361-1405-4d84-9f68-d54dcffc9180
Baseline: 0fbbe1479d09b164c6c2d15401ac7656e0cd8a38
Comparison: 8ad27e32850a98d350ee31d94a1414df1d74105d
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_regex_logs2metric_ddmetrics 145.67KiB/CPU-s 3.91 100.00% 3.64MiB/CPU-s 475.73KiB/CPU-s 6.14KiB/CPU-s 0.0 0.127618 3.78MiB/CPU-s 456.91KiB/CPU-s 5.9KiB/CPU-s 0.0 0.11796 True False
http_text_to_http_json 358.94KiB/CPU-s 1.47 100.00% 23.89MiB/CPU-s 587.49KiB/CPU-s 7.58KiB/CPU-s 0.0 0.024013 24.24MiB/CPU-s 662.15KiB/CPU-s 8.54KiB/CPU-s 0.0 0.026673 False False
syslog_loki 102.34KiB/CPU-s 1.14 100.00% 8.8MiB/CPU-s 293.45KiB/CPU-s 3.79KiB/CPU-s 0.0 0.032546 8.9MiB/CPU-s 151.37KiB/CPU-s 1.95KiB/CPU-s 0.0 0.0166 False False
syslog_humio_logs 103.9KiB/CPU-s 1.1 100.00% 9.2MiB/CPU-s 230.38KiB/CPU-s 2.97KiB/CPU-s 0.0 0.024445 9.3MiB/CPU-s 106.78KiB/CPU-s 1.38KiB/CPU-s 0.0 0.011207 False False
datadog_agent_remap_datadog_logs_acks 260.1KiB/CPU-s 0.77 100.00% 32.82MiB/CPU-s 1.48MiB/CPU-s 19.53KiB/CPU-s 0.0 0.045025 33.08MiB/CPU-s 1.07MiB/CPU-s 14.16KiB/CPU-s 0.0 0.032398 False False
syslog_log2metric_splunk_hec_metrics 66.52KiB/CPU-s 0.7 100.00% 9.3MiB/CPU-s 471.25KiB/CPU-s 6.08KiB/CPU-s 0.0 0.04946 9.37MiB/CPU-s 512.87KiB/CPU-s 6.62KiB/CPU-s 0.0 0.053454 False False
http_to_http_acks 20.16KiB/CPU-s 0.38 30.54% 5.25MiB/CPU-s 2.72MiB/CPU-s 35.94KiB/CPU-s 0.0 0.517951 5.27MiB/CPU-s 2.78MiB/CPU-s 36.68KiB/CPU-s 0.0 0.526741 True False
syslog_splunk_hec_logs 35.22KiB/CPU-s 0.37 100.00% 9.19MiB/CPU-s 123.85KiB/CPU-s 1.6KiB/CPU-s 0.0 0.013165 9.22MiB/CPU-s 175.6KiB/CPU-s 2.27KiB/CPU-s 0.0 0.018596 False False
otlp_grpc_to_blackhole 3.8KiB/CPU-s 0.36 100.00% 1.04MiB/CPU-s 38.7KiB/CPU-s 511.58B/CPU-s 0.0 0.036335 1.04MiB/CPU-s 48.12KiB/CPU-s 635.85B/CPU-s 0.0 0.045015 False False
otlp_http_to_blackhole 5.17KiB/CPU-s 0.33 98.58% 1.52MiB/CPU-s 112.91KiB/CPU-s 1.46KiB/CPU-s 0.0 0.072338 1.53MiB/CPU-s 117.75KiB/CPU-s 1.52KiB/CPU-s 0.0 0.07519 False False
socket_to_socket_blackhole 40.3KiB/CPU-s 0.3 100.00% 13.24MiB/CPU-s 224.3KiB/CPU-s 2.89KiB/CPU-s 0.0 0.016539 13.28MiB/CPU-s 78.24KiB/CPU-s 1.01KiB/CPU-s 0.0 0.005752 False False
enterprise_http_to_http 14.56KiB/CPU-s 0.1 99.69% 13.61MiB/CPU-s 351.88KiB/CPU-s 4.54KiB/CPU-s 0.0 0.025247 13.62MiB/CPU-s 148.44KiB/CPU-s 1.92KiB/CPU-s 0.0 0.010639 False False
datadog_agent_remap_datadog_logs 23.19KiB/CPU-s 0.07 69.47% 34.15MiB/CPU-s 1.16MiB/CPU-s 15.37KiB/CPU-s 0.0 0.034078 34.17MiB/CPU-s 1.26MiB/CPU-s 16.59KiB/CPU-s 0.0 0.036756 False False
splunk_hec_to_splunk_hec_logs_noack 3.32KiB/CPU-s 0.02 62.29% 13.62MiB/CPU-s 220.85KiB/CPU-s 2.85KiB/CPU-s 0.0 0.015832 13.63MiB/CPU-s 190.21KiB/CPU-s 2.46KiB/CPU-s 0.0 0.013632 False False
splunk_hec_indexer_ack_blackhole 1.34KiB/CPU-s 0.01 22.68% 13.62MiB/CPU-s 257.2KiB/CPU-s 3.32KiB/CPU-s 0.0 0.018445 13.62MiB/CPU-s 251.58KiB/CPU-s 3.25KiB/CPU-s 0.0 0.01804 False False
splunk_hec_to_splunk_hec_logs_acks 1.23KiB/CPU-s 0.01 15.83% 13.62MiB/CPU-s 328.94KiB/CPU-s 4.24KiB/CPU-s 0.0 0.023589 13.62MiB/CPU-s 345.35KiB/CPU-s 4.46KiB/CPU-s 0.0 0.024764 False False
fluent_elasticsearch 368.38B/CPU-s 0.0 49.16% 45.41MiB/CPU-s 30.07KiB/CPU-s 393.2B/CPU-s 0.0 0.000646 45.41MiB/CPU-s 30.19KiB/CPU-s 394.56B/CPU-s 0.0 0.000649 False False
http_to_http_json -1.01KiB/CPU-s -0.01 20.74% 13.62MiB/CPU-s 209.54KiB/CPU-s 2.7KiB/CPU-s 0.0 0.015019 13.62MiB/CPU-s 211.14KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015136 False False
file_to_blackhole -10.17KiB/CPU-s -0.02 36.19% 54.5MiB/CPU-s 1.15MiB/CPU-s 15.19KiB/CPU-s 0.0 0.021097 54.49MiB/CPU-s 1.17MiB/CPU-s 15.4KiB/CPU-s 0.0 0.021406 False False
http_to_http_noack -6.44KiB/CPU-s -0.05 73.24% 13.62MiB/CPU-s 284.35KiB/CPU-s 3.67KiB/CPU-s 0.0 0.020393 13.61MiB/CPU-s 349.17KiB/CPU-s 4.51KiB/CPU-s 0.0 0.025053 False False
datadog_agent_remap_blackhole_acks -308.44KiB/CPU-s -0.98 100.00% 30.69MiB/CPU-s 1.76MiB/CPU-s 23.32KiB/CPU-s 0.0 0.057458 30.39MiB/CPU-s 2.59MiB/CPU-s 34.3KiB/CPU-s 0.0 0.085351 False False
syslog_log2metric_humio_metrics -90.48KiB/CPU-s -1.41 100.00% 6.28MiB/CPU-s 95.49KiB/CPU-s 1.23KiB/CPU-s 0.0 0.014852 6.19MiB/CPU-s 208.5KiB/CPU-s 2.69KiB/CPU-s 0.0 0.032893 False False
datadog_agent_remap_blackhole -494.52KiB/CPU-s -1.54 100.00% 31.44MiB/CPU-s 760.69KiB/CPU-s 9.82KiB/CPU-s 0.0 0.023622 30.96MiB/CPU-s 1.55MiB/CPU-s 20.42KiB/CPU-s 0.0 0.049899 False False
splunk_hec_route_s3 -203.96KiB/CPU-s -1.68 100.00% 11.82MiB/CPU-s 524.31KiB/CPU-s 6.76KiB/CPU-s 0.0 0.043311 11.62MiB/CPU-s 718.18KiB/CPU-s 9.27KiB/CPU-s 0.0 0.060343 False False

github-actions[bot] avatar Jan 17 '23 16:01 github-actions[bot]