vector
vector copied to clipboard
vectordotdev
feat(aws_kinesis_firehose_source): Configure multiple access keys
Deploy Preview for vrl-playground canceled.
| Name | Link |
|---|---|
| Latest commit | f63bf6da9b043819c4540b043c22e0b5bcab1b91 |
| Latest deploy log | https://app.netlify.com/sites/vrl-playground/deploys/63c07bd53c637b00089ad95f |
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
Deploy Preview for vector-project ready!
| Name | Link |
|---|---|
| Latest commit | f63bf6da9b043819c4540b043c22e0b5bcab1b91 |
| Latest deploy log | https://app.netlify.com/sites/vector-project/deploys/63c07bd59c8d7d0008e001d8 |
| Deploy Preview | https://deploy-preview-15828--vector-project.netlify.app |
| Preview on mobile | Toggle QR Code...Use your smartphone camera to open QR code link. |
To edit notification comments on pull requests, go to your Netlify site settings.
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.
:white_check_mark: dizlv
:white_check_mark: spencergilbert
:x: Dmitrijs Zubriks
Dmitrijs Zubriks seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
Regression Test Results
Run ID: 862d0391-ceea-4c63-9560-f0372b61dc16
Baseline: f1daacb332973dc9433e757bfa8a3f253c72d0b0
Comparison: 9a4804c552539d965d5684e8088fe9000f11f0f5
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparision SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| syslog_log2metric_splunk_hec_metrics | -483.13KiB/CPU-s | -5.06 | 100.00% |
| syslog_humio_logs | -475.56KiB/CPU-s | -5.09 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_regex_logs2metric_ddmetrics | 28.15KiB/CPU-s | 0.78 | 99.83% | 3.53MiB/CPU-s | 479.14KiB/CPU-s | 6.18KiB/CPU-s | 0.0 | 0.132592 | 3.56MiB/CPU-s | 500.86KiB/CPU-s | 6.47KiB/CPU-s | 0.0 | 0.13753 | True | False |
| datadog_agent_remap_datadog_logs | 46.05KiB/CPU-s | 0.13 | 91.79% | 33.61MiB/CPU-s | 1.51MiB/CPU-s | 20.0KiB/CPU-s | 0.0 | 0.045042 | 33.66MiB/CPU-s | 1.31MiB/CPU-s | 17.36KiB/CPU-s | 0.0 | 0.039048 | False | False |
| enterprise_http_to_http | 10.55KiB/CPU-s | 0.08 | 98.27% | 13.61MiB/CPU-s | 309.15KiB/CPU-s | 3.99KiB/CPU-s | 0.0 | 0.022174 | 13.62MiB/CPU-s | 148.96KiB/CPU-s | 1.92KiB/CPU-s | 0.0 | 0.010677 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 3.58KiB/CPU-s | 0.03 | 65.47% | 13.62MiB/CPU-s | 221.32KiB/CPU-s | 2.86KiB/CPU-s | 0.0 | 0.015865 | 13.63MiB/CPU-s | 192.86KiB/CPU-s | 2.49KiB/CPU-s | 0.0 | 0.013821 | False | False |
| file_to_blackhole | 5.97KiB/CPU-s | 0.01 | 22.65% | 54.49MiB/CPU-s | 1.07MiB/CPU-s | 14.1KiB/CPU-s | 0.0 | 0.019591 | 54.5MiB/CPU-s | 1.15MiB/CPU-s | 15.24KiB/CPU-s | 0.0 | 0.021173 | False | False |
| http_to_http_json | -58.23B/CPU-s | -0.0 | 1.19% | 13.62MiB/CPU-s | 212.48KiB/CPU-s | 2.74KiB/CPU-s | 0.0 | 0.015232 | 13.62MiB/CPU-s | 206.38KiB/CPU-s | 2.66KiB/CPU-s | 0.0 | 0.014795 | False | False |
| splunk_hec_indexer_ack_blackhole | -10.98B/CPU-s | -0.0 | 0.18% | 13.62MiB/CPU-s | 253.5KiB/CPU-s | 3.27KiB/CPU-s | 0.0 | 0.018179 | 13.62MiB/CPU-s | 255.1KiB/CPU-s | 3.29KiB/CPU-s | 0.0 | 0.018294 | False | False |
| fluent_elasticsearch | -3.83KiB/CPU-s | -0.01 | 68.25% | 45.41MiB/CPU-s | 29.79KiB/CPU-s | 389.44B/CPU-s | 0.0 | 0.000641 | 45.41MiB/CPU-s | 298.33KiB/CPU-s | 3.81KiB/CPU-s | 0.0 | 0.006415 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -1.13KiB/CPU-s | -0.01 | 14.92% | 13.62MiB/CPU-s | 322.37KiB/CPU-s | 4.16KiB/CPU-s | 0.0 | 0.023114 | 13.62MiB/CPU-s | 335.17KiB/CPU-s | 4.32KiB/CPU-s | 0.0 | 0.024033 | False | False |
| http_to_http_noack | -6.36KiB/CPU-s | -0.05 | 76.28% | 13.62MiB/CPU-s | 250.38KiB/CPU-s | 3.23KiB/CPU-s | 0.0 | 0.017953 | 13.61MiB/CPU-s | 333.32KiB/CPU-s | 4.3KiB/CPU-s | 0.0 | 0.023912 | False | False |
| datadog_agent_remap_blackhole_acks | -186.4KiB/CPU-s | -0.59 | 100.00% | 30.79MiB/CPU-s | 1.83MiB/CPU-s | 24.18KiB/CPU-s | 0.0 | 0.059389 | 30.61MiB/CPU-s | 1.35MiB/CPU-s | 17.8KiB/CPU-s | 0.0 | 0.044008 | False | False |
| syslog_log2metric_humio_metrics | -42.23KiB/CPU-s | -0.68 | 100.00% | 6.08MiB/CPU-s | 338.98KiB/CPU-s | 4.38KiB/CPU-s | 0.0 | 0.054483 | 6.03MiB/CPU-s | 215.5KiB/CPU-s | 2.78KiB/CPU-s | 0.0 | 0.034874 | False | False |
| http_to_http_acks | -46.52KiB/CPU-s | -0.86 | 62.78% | 5.27MiB/CPU-s | 2.81MiB/CPU-s | 37.18KiB/CPU-s | 0.0 | 0.53398 | 5.22MiB/CPU-s | 2.77MiB/CPU-s | 36.55KiB/CPU-s | 0.0 | 0.529498 | True | False |
| splunk_hec_route_s3 | -147.03KiB/CPU-s | -1.21 | 100.00% | 11.86MiB/CPU-s | 525.58KiB/CPU-s | 6.78KiB/CPU-s | 0.0 | 0.04328 | 11.71MiB/CPU-s | 601.3KiB/CPU-s | 7.76KiB/CPU-s | 0.0 | 0.050122 | False | False |
| datadog_agent_remap_blackhole | -476.95KiB/CPU-s | -1.5 | 100.00% | 31.07MiB/CPU-s | 1.24MiB/CPU-s | 16.4KiB/CPU-s | 0.0 | 0.039926 | 30.6MiB/CPU-s | 1.06MiB/CPU-s | 13.99KiB/CPU-s | 0.0 | 0.034599 | False | False |
| otlp_http_to_blackhole | -30.9KiB/CPU-s | -1.94 | 100.00% | 1.55MiB/CPU-s | 103.39KiB/CPU-s | 1.33KiB/CPU-s | 0.0 | 0.065063 | 1.52MiB/CPU-s | 118.88KiB/CPU-s | 1.53KiB/CPU-s | 0.0 | 0.076299 | False | False |
| socket_to_socket_blackhole | -310.25KiB/CPU-s | -2.25 | 100.00% | 13.45MiB/CPU-s | 357.77KiB/CPU-s | 4.62KiB/CPU-s | 0.0 | 0.025965 | 13.15MiB/CPU-s | 188.16KiB/CPU-s | 2.43KiB/CPU-s | 0.0 | 0.01397 | False | False |
| datadog_agent_remap_datadog_logs_acks | -792.94KiB/CPU-s | -2.28 | 100.00% | 33.9MiB/CPU-s | 1.36MiB/CPU-s | 17.97KiB/CPU-s | 0.0 | 0.040127 | 33.12MiB/CPU-s | 1.66MiB/CPU-s | 21.93KiB/CPU-s | 0.0 | 0.050084 | False | False |
| syslog_splunk_hec_logs | -297.82KiB/CPU-s | -3.22 | 100.00% | 9.03MiB/CPU-s | 217.11KiB/CPU-s | 2.8KiB/CPU-s | 0.0 | 0.023485 | 8.74MiB/CPU-s | 216.69KiB/CPU-s | 2.8KiB/CPU-s | 0.0 | 0.02422 | False | False |
| http_text_to_http_json | -843.06KiB/CPU-s | -3.34 | 100.00% | 24.65MiB/CPU-s | 792.15KiB/CPU-s | 10.23KiB/CPU-s | 0.0 | 0.031374 | 23.83MiB/CPU-s | 559.68KiB/CPU-s | 7.22KiB/CPU-s | 0.0 | 0.022933 | False | False |
| syslog_loki | -306.74KiB/CPU-s | -3.49 | 100.00% | 8.57MiB/CPU-s | 273.01KiB/CPU-s | 3.52KiB/CPU-s | 0.0 | 0.03109 | 8.28MiB/CPU-s | 277.4KiB/CPU-s | 3.58KiB/CPU-s | 0.0 | 0.032734 | False | False |
| otlp_grpc_to_blackhole | -40.18KiB/CPU-s | -3.72 | 100.00% | 1.06MiB/CPU-s | 32.07KiB/CPU-s | 423.91B/CPU-s | 0.0 | 0.029661 | 1.02MiB/CPU-s | 53.16KiB/CPU-s | 702.45B/CPU-s | 0.0 | 0.051073 | False | False |
| syslog_log2metric_splunk_hec_metrics | -483.13KiB/CPU-s | -5.06 | 100.00% | 9.32MiB/CPU-s | 337.15KiB/CPU-s | 4.35KiB/CPU-s | 0.0 | 0.035328 | 8.85MiB/CPU-s | 283.19KiB/CPU-s | 3.66KiB/CPU-s | 0.0 | 0.031257 | False | False |
| syslog_humio_logs | -475.56KiB/CPU-s | -5.09 | 100.00% | 9.13MiB/CPU-s | 218.64KiB/CPU-s | 2.82KiB/CPU-s | 0.0 | 0.023397 | 8.66MiB/CPU-s | 166.16KiB/CPU-s | 2.15KiB/CPU-s | 0.0 | 0.018734 | False | False |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Just checking @dizlv - this is still WIP and doesn't necessarily need a review until you Ready it?
@spencergilbert I'm currently trying to compile binary to test this feature in my setup, but I do not plan to make any other changes, except code for passing around strings in tests. Probably you want to take a look on it and assist me a little – I had no time to review documentation yet on cloning & referencing in similar codes?
Regression Test Results
Run ID: 149e6c36-26b1-4f73-8af6-68c562150ae6
Baseline: d04826e4c2c7ce7f18d293d4d35c85f161b007d4
Comparison: e174a765fb08b5c74123cc65c57cfd5eb430afeb
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparision SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_regex_logs2metric_ddmetrics | 48.51KiB/CPU-s | 1.37 | 100.00% | 3.46MiB/CPU-s | 289.3KiB/CPU-s | 3.73KiB/CPU-s | 0.0 | 0.081588 | 3.51MiB/CPU-s | 300.71KiB/CPU-s | 3.88KiB/CPU-s | 0.0 | 0.083662 | False | False |
| http_to_http_acks | 26.17KiB/CPU-s | 0.49 | 38.89% | 5.26MiB/CPU-s | 2.75MiB/CPU-s | 36.39KiB/CPU-s | 0.0 | 0.523533 | 5.29MiB/CPU-s | 2.75MiB/CPU-s | 36.39KiB/CPU-s | 0.0 | 0.521101 | True | False |
| enterprise_http_to_http | 10.44KiB/CPU-s | 0.07 | 98.07% | 13.61MiB/CPU-s | 310.41KiB/CPU-s | 4.01KiB/CPU-s | 0.0 | 0.022265 | 13.62MiB/CPU-s | 151.82KiB/CPU-s | 1.96KiB/CPU-s | 0.0 | 0.010881 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 3.74KiB/CPU-s | 0.03 | 68.04% | 13.62MiB/CPU-s | 218.73KiB/CPU-s | 2.82KiB/CPU-s | 0.0 | 0.01568 | 13.63MiB/CPU-s | 191.6KiB/CPU-s | 2.47KiB/CPU-s | 0.0 | 0.013731 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 4.25KiB/CPU-s | 0.03 | 47.40% | 13.61MiB/CPU-s | 371.87KiB/CPU-s | 4.8KiB/CPU-s | 0.0 | 0.026673 | 13.62MiB/CPU-s | 362.32KiB/CPU-s | 4.68KiB/CPU-s | 0.0 | 0.02598 | False | False |
| http_to_http_noack | 4.56KiB/CPU-s | 0.03 | 61.78% | 13.61MiB/CPU-s | 310.84KiB/CPU-s | 4.01KiB/CPU-s | 0.0 | 0.022296 | 13.62MiB/CPU-s | 258.79KiB/CPU-s | 3.34KiB/CPU-s | 0.0 | 0.018556 | False | False |
| splunk_hec_indexer_ack_blackhole | 958.17B/CPU-s | 0.01 | 16.05% | 13.62MiB/CPU-s | 255.97KiB/CPU-s | 3.3KiB/CPU-s | 0.0 | 0.018357 | 13.62MiB/CPU-s | 250.37KiB/CPU-s | 3.23KiB/CPU-s | 0.0 | 0.017954 | False | False |
| fluent_elasticsearch | 348.63B/CPU-s | 0.0 | 46.86% | 45.41MiB/CPU-s | 30.3KiB/CPU-s | 395.95B/CPU-s | 0.0 | 0.000651 | 45.41MiB/CPU-s | 29.98KiB/CPU-s | 391.71B/CPU-s | 0.0 | 0.000645 | False | False |
| file_to_blackhole | -5.59KiB/CPU-s | -0.01 | 21.53% | 54.5MiB/CPU-s | 1.09MiB/CPU-s | 14.43KiB/CPU-s | 0.0 | 0.02004 | 54.49MiB/CPU-s | 1.1MiB/CPU-s | 14.53KiB/CPU-s | 0.0 | 0.0202 | False | False |
| http_to_http_json | -4.88KiB/CPU-s | -0.03 | 79.35% | 13.62MiB/CPU-s | 210.34KiB/CPU-s | 2.71KiB/CPU-s | 0.0 | 0.015078 | 13.62MiB/CPU-s | 213.32KiB/CPU-s | 2.75KiB/CPU-s | 0.0 | 0.015297 | False | False |
| otlp_http_to_blackhole | -1.28KiB/CPU-s | -0.08 | 42.11% | 1.52MiB/CPU-s | 124.51KiB/CPU-s | 1.61KiB/CPU-s | 0.0 | 0.080189 | 1.51MiB/CPU-s | 127.89KiB/CPU-s | 1.65KiB/CPU-s | 0.0 | 0.082436 | False | False |
| datadog_agent_remap_blackhole_acks | -34.11KiB/CPU-s | -0.11 | 85.91% | 30.52MiB/CPU-s | 1.15MiB/CPU-s | 15.16KiB/CPU-s | 0.0 | 0.037569 | 30.49MiB/CPU-s | 1.32MiB/CPU-s | 17.51KiB/CPU-s | 0.0 | 0.043442 | False | False |
| datadog_agent_remap_blackhole | -222.25KiB/CPU-s | -0.7 | 100.00% | 30.8MiB/CPU-s | 927.3KiB/CPU-s | 11.98KiB/CPU-s | 0.0 | 0.029404 | 30.58MiB/CPU-s | 735.72KiB/CPU-s | 9.5KiB/CPU-s | 0.0 | 0.023494 | False | False |
| datadog_agent_remap_datadog_logs_acks | -422.37KiB/CPU-s | -1.23 | 100.00% | 33.49MiB/CPU-s | 1.44MiB/CPU-s | 18.98KiB/CPU-s | 0.0 | 0.042906 | 33.08MiB/CPU-s | 1.03MiB/CPU-s | 13.68KiB/CPU-s | 0.0 | 0.031279 | False | False |
| splunk_hec_route_s3 | -178.45KiB/CPU-s | -1.49 | 100.00% | 11.73MiB/CPU-s | 569.28KiB/CPU-s | 7.34KiB/CPU-s | 0.0 | 0.047371 | 11.56MiB/CPU-s | 593.75KiB/CPU-s | 7.66KiB/CPU-s | 0.0 | 0.050152 | False | False |
| datadog_agent_remap_datadog_logs | -529.81KiB/CPU-s | -1.53 | 100.00% | 33.88MiB/CPU-s | 1.24MiB/CPU-s | 16.4KiB/CPU-s | 0.0 | 0.036632 | 33.37MiB/CPU-s | 1.15MiB/CPU-s | 15.23KiB/CPU-s | 0.0 | 0.034549 | False | False |
| syslog_humio_logs | -193.83KiB/CPU-s | -2.16 | 100.00% | 8.76MiB/CPU-s | 491.86KiB/CPU-s | 6.35KiB/CPU-s | 0.0 | 0.054799 | 8.58MiB/CPU-s | 273.1KiB/CPU-s | 3.53KiB/CPU-s | 0.0 | 0.031098 | False | False |
| otlp_grpc_to_blackhole | -30.63KiB/CPU-s | -2.85 | 100.00% | 1.05MiB/CPU-s | 41.72KiB/CPU-s | 551.39B/CPU-s | 0.0 | 0.038775 | 1.02MiB/CPU-s | 52.91KiB/CPU-s | 699.07B/CPU-s | 0.0 | 0.050623 | False | False |
| syslog_log2metric_humio_metrics | -174.7KiB/CPU-s | -2.92 | 100.00% | 5.85MiB/CPU-s | 150.98KiB/CPU-s | 1.95KiB/CPU-s | 0.0 | 0.025218 | 5.68MiB/CPU-s | 257.3KiB/CPU-s | 3.32KiB/CPU-s | 0.0 | 0.04427 | False | False |
| socket_to_socket_blackhole | -427.46KiB/CPU-s | -3.09 | 100.00% | 13.5MiB/CPU-s | 305.29KiB/CPU-s | 3.94KiB/CPU-s | 0.0 | 0.02209 | 13.08MiB/CPU-s | 271.46KiB/CPU-s | 3.51KiB/CPU-s | 0.0 | 0.020269 | False | False |
| http_text_to_http_json | -954.81KiB/CPU-s | -3.8 | 100.00% | 24.56MiB/CPU-s | 576.25KiB/CPU-s | 7.44KiB/CPU-s | 0.0 | 0.022916 | 23.62MiB/CPU-s | 489.38KiB/CPU-s | 6.32KiB/CPU-s | 0.0 | 0.020229 | False | False |
| syslog_splunk_hec_logs | -394.89KiB/CPU-s | -4.32 | 100.00% | 8.94MiB/CPU-s | 188.45KiB/CPU-s | 2.43KiB/CPU-s | 0.0 | 0.020595 | 8.55MiB/CPU-s | 258.85KiB/CPU-s | 3.34KiB/CPU-s | 0.0 | 0.029565 | False | False |
| syslog_loki | -402.49KiB/CPU-s | -4.57 | 100.00% | 8.59MiB/CPU-s | 430.67KiB/CPU-s | 5.56KiB/CPU-s | 0.0 | 0.048937 | 8.2MiB/CPU-s | 295.02KiB/CPU-s | 3.81KiB/CPU-s | 0.0 | 0.03513 | False | False |
| syslog_log2metric_splunk_hec_metrics | -442.16KiB/CPU-s | -4.66 | 100.00% | 9.26MiB/CPU-s | 245.4KiB/CPU-s | 3.17KiB/CPU-s | 0.0 | 0.025888 | 8.82MiB/CPU-s | 328.6KiB/CPU-s | 4.24KiB/CPU-s | 0.0 | 0.036362 | False | False |
@spencergilbert hey, I have added deprecation notes and about to test this with actual Firehose setup. Other than that and String cloning in tests I think I don't have any new changes and it should be ready to move from draft to the PR.
Update:
Tested access keys with real Firehose, it now is capable to handle multiple keys.
Awesome! I've added it to be todos, hopefully will get to it by tomorrow.
Regression Test Results
Run ID: 49a86678-7d42-4006-8917-a070042552ff
Baseline: cbe52f0c5bd7ed4161f7b8fa36ac88ec27b2bffa
Comparison: b219daf9cb077a3a45f6ccaadfa4f829292a51a5
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| http_text_to_http_json | -1.36MiB/CPU-s | -5.43 | 100.00% |
| syslog_log2metric_splunk_hec_metrics | -590.3KiB/CPU-s | -6.09 | 100.00% |
| syslog_loki | -723.86KiB/CPU-s | -7.89 | 100.00% |
| otlp_http_to_blackhole | -148.87KiB/CPU-s | -9.57 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole_acks | 888.84KiB/CPU-s | 2.85 | 100.00% | 30.45MiB/CPU-s | 1.78MiB/CPU-s | 23.57KiB/CPU-s | 0.0 | 0.058553 | 31.31MiB/CPU-s | 926.83KiB/CPU-s | 11.96KiB/CPU-s | 0.0 | 0.028902 | False | False |
| http_to_http_json | 10.45KiB/CPU-s | 0.07 | 98.95% | 13.61MiB/CPU-s | 240.76KiB/CPU-s | 3.11KiB/CPU-s | 0.0 | 0.017271 | 13.62MiB/CPU-s | 205.51KiB/CPU-s | 2.65KiB/CPU-s | 0.0 | 0.014732 | False | False |
| enterprise_http_to_http | 6.37KiB/CPU-s | 0.05 | 76.40% | 13.61MiB/CPU-s | 332.16KiB/CPU-s | 4.29KiB/CPU-s | 0.0 | 0.023829 | 13.62MiB/CPU-s | 251.66KiB/CPU-s | 3.25KiB/CPU-s | 0.0 | 0.018045 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 5.62KiB/CPU-s | 0.04 | 84.70% | 13.62MiB/CPU-s | 236.94KiB/CPU-s | 3.06KiB/CPU-s | 0.0 | 0.016987 | 13.63MiB/CPU-s | 191.97KiB/CPU-s | 2.48KiB/CPU-s | 0.0 | 0.013758 | False | False |
| http_to_http_noack | 3.88KiB/CPU-s | 0.03 | 45.64% | 13.61MiB/CPU-s | 368.16KiB/CPU-s | 4.75KiB/CPU-s | 0.0 | 0.026418 | 13.61MiB/CPU-s | 331.04KiB/CPU-s | 4.27KiB/CPU-s | 0.0 | 0.023748 | False | False |
| splunk_hec_indexer_ack_blackhole | 927.21B/CPU-s | 0.01 | 15.79% | 13.62MiB/CPU-s | 252.48KiB/CPU-s | 3.26KiB/CPU-s | 0.0 | 0.018105 | 13.62MiB/CPU-s | 245.79KiB/CPU-s | 3.17KiB/CPU-s | 0.0 | 0.017624 | False | False |
| fluent_elasticsearch | -32.75B/CPU-s | -0.0 | 4.68% | 45.41MiB/CPU-s | 30.29KiB/CPU-s | 395.78B/CPU-s | 0.0 | 0.000651 | 45.41MiB/CPU-s | 30.1KiB/CPU-s | 393.46B/CPU-s | 0.0 | 0.000647 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -1.35KiB/CPU-s | -0.01 | 16.51% | 13.61MiB/CPU-s | 348.73KiB/CPU-s | 4.5KiB/CPU-s | 0.0 | 0.025012 | 13.61MiB/CPU-s | 359.37KiB/CPU-s | 4.64KiB/CPU-s | 0.0 | 0.025778 | False | False |
| file_to_blackhole | -21.33KiB/CPU-s | -0.04 | 66.25% | 54.49MiB/CPU-s | 1.11MiB/CPU-s | 14.7KiB/CPU-s | 0.0 | 0.020422 | 54.47MiB/CPU-s | 1.26MiB/CPU-s | 16.69KiB/CPU-s | 0.0 | 0.023214 | False | False |
| datadog_agent_remap_datadog_logs_acks | -133.35KiB/CPU-s | -0.39 | 100.00% | 33.1MiB/CPU-s | 1.8MiB/CPU-s | 23.76KiB/CPU-s | 0.0 | 0.054335 | 32.97MiB/CPU-s | 1.12MiB/CPU-s | 14.76KiB/CPU-s | 0.0 | 0.03388 | False | False |
| datadog_agent_remap_blackhole | -273.79KiB/CPU-s | -0.85 | 100.00% | 31.34MiB/CPU-s | 1.1MiB/CPU-s | 14.56KiB/CPU-s | 0.0 | 0.035143 | 31.07MiB/CPU-s | 952.31KiB/CPU-s | 12.29KiB/CPU-s | 0.0 | 0.029929 | False | False |
| syslog_regex_logs2metric_ddmetrics | -38.49KiB/CPU-s | -1.0 | 100.00% | 3.76MiB/CPU-s | 482.5KiB/CPU-s | 6.23KiB/CPU-s | 0.0 | 0.125161 | 3.73MiB/CPU-s | 437.71KiB/CPU-s | 5.65KiB/CPU-s | 0.0 | 0.114688 | True | False |
| splunk_hec_route_s3 | -126.54KiB/CPU-s | -1.08 | 100.00% | 11.41MiB/CPU-s | 667.44KiB/CPU-s | 8.61KiB/CPU-s | 0.0 | 0.05713 | 11.28MiB/CPU-s | 766.4KiB/CPU-s | 9.89KiB/CPU-s | 0.0 | 0.066319 | False | False |
| http_to_http_acks | -96.88KiB/CPU-s | -1.78 | 93.90% | 5.33MiB/CPU-s | 2.73MiB/CPU-s | 36.11KiB/CPU-s | 0.0 | 0.513074 | 5.23MiB/CPU-s | 2.8MiB/CPU-s | 37.0KiB/CPU-s | 0.0 | 0.535193 | True | False |
| datadog_agent_remap_datadog_logs | -652.52KiB/CPU-s | -1.86 | 100.00% | 34.23MiB/CPU-s | 1.43MiB/CPU-s | 18.89KiB/CPU-s | 0.0 | 0.041769 | 33.59MiB/CPU-s | 1.21MiB/CPU-s | 16.02KiB/CPU-s | 0.0 | 0.036101 | False | False |
| otlp_grpc_to_blackhole | -23.36KiB/CPU-s | -2.2 | 100.00% | 1.04MiB/CPU-s | 49.73KiB/CPU-s | 657.23B/CPU-s | 0.0 | 0.046813 | 1.01MiB/CPU-s | 54.22KiB/CPU-s | 716.4B/CPU-s | 0.0 | 0.052189 | False | False |
| socket_to_socket_blackhole | -352.96KiB/CPU-s | -2.52 | 100.00% | 13.68MiB/CPU-s | 216.91KiB/CPU-s | 2.8KiB/CPU-s | 0.0 | 0.015485 | 13.33MiB/CPU-s | 145.02KiB/CPU-s | 1.87KiB/CPU-s | 0.0 | 0.01062 | False | False |
| syslog_log2metric_humio_metrics | -212.14KiB/CPU-s | -3.37 | 100.00% | 6.15MiB/CPU-s | 192.94KiB/CPU-s | 2.49KiB/CPU-s | 0.0 | 0.030615 | 5.95MiB/CPU-s | 263.32KiB/CPU-s | 3.4KiB/CPU-s | 0.0 | 0.043237 | False | False |
| syslog_splunk_hec_logs | -316.16KiB/CPU-s | -3.42 | 100.00% | 9.02MiB/CPU-s | 294.2KiB/CPU-s | 3.8KiB/CPU-s | 0.0 | 0.031863 | 8.71MiB/CPU-s | 177.39KiB/CPU-s | 2.29KiB/CPU-s | 0.0 | 0.019893 | False | False |
| syslog_humio_logs | -360.76KiB/CPU-s | -3.91 | 100.00% | 9.02MiB/CPU-s | 426.36KiB/CPU-s | 5.5KiB/CPU-s | 0.0 | 0.046148 | 8.67MiB/CPU-s | 150.13KiB/CPU-s | 1.94KiB/CPU-s | 0.0 | 0.01691 | False | False |
| http_text_to_http_json | -1.36MiB/CPU-s | -5.43 | 100.00% | 25.03MiB/CPU-s | 717.08KiB/CPU-s | 9.26KiB/CPU-s | 0.0 | 0.027976 | 23.67MiB/CPU-s | 750.41KiB/CPU-s | 9.68KiB/CPU-s | 0.0 | 0.030957 | False | False |
| syslog_log2metric_splunk_hec_metrics | -590.3KiB/CPU-s | -6.09 | 100.00% | 9.47MiB/CPU-s | 377.96KiB/CPU-s | 4.88KiB/CPU-s | 0.0 | 0.038984 | 8.89MiB/CPU-s | 292.96KiB/CPU-s | 3.78KiB/CPU-s | 0.0 | 0.032176 | False | False |
| syslog_loki | -723.86KiB/CPU-s | -7.89 | 100.00% | 8.96MiB/CPU-s | 183.37KiB/CPU-s | 2.37KiB/CPU-s | 0.0 | 0.019992 | 8.25MiB/CPU-s | 250.31KiB/CPU-s | 3.23KiB/CPU-s | 0.0 | 0.029628 | False | False |
| otlp_http_to_blackhole | -148.87KiB/CPU-s | -9.57 | 100.00% | 1.52MiB/CPU-s | 125.22KiB/CPU-s | 1.62KiB/CPU-s | 0.0 | 0.080458 | 1.37MiB/CPU-s | 448.9KiB/CPU-s | 5.79KiB/CPU-s | 0.0 | 0.318947 | True | False |
Regression Test Results
Run ID: c31180fa-bce9-4b25-a986-65bc3a1dec17
Baseline: bd1ccc0b9d45d91dee21960f5b39e9c7ae273f51
Comparison: a0b6f502f13a32b39938da6a92c6d8988d52163e
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| syslog_log2metric_splunk_hec_metrics | -478.75KiB/CPU-s | -5.05 | 100.00% |
| syslog_splunk_hec_logs | -538.21KiB/CPU-s | -5.77 | 100.00% |
| syslog_loki | -573.25KiB/CPU-s | -6.33 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole_acks | 98.23KiB/CPU-s | 0.31 | 100.00% | 30.96MiB/CPU-s | 1.07MiB/CPU-s | 14.2KiB/CPU-s | 0.0 | 0.034681 | 31.06MiB/CPU-s | 1.02MiB/CPU-s | 13.54KiB/CPU-s | 0.0 | 0.032963 | False | False |
| splunk_hec_route_s3 | 28.88KiB/CPU-s | 0.25 | 97.74% | 11.42MiB/CPU-s | 714.44KiB/CPU-s | 9.22KiB/CPU-s | 0.0 | 0.061067 | 11.45MiB/CPU-s | 673.16KiB/CPU-s | 8.69KiB/CPU-s | 0.0 | 0.057397 | False | False |
| syslog_log2metric_humio_metrics | 14.83KiB/CPU-s | 0.24 | 99.59% | 5.91MiB/CPU-s | 289.79KiB/CPU-s | 3.74KiB/CPU-s | 0.0 | 0.04785 | 5.93MiB/CPU-s | 275.54KiB/CPU-s | 3.56KiB/CPU-s | 0.0 | 0.045385 | False | False |
| http_to_http_json | 30.88KiB/CPU-s | 0.22 | 100.00% | 13.59MiB/CPU-s | 285.31KiB/CPU-s | 3.68KiB/CPU-s | 0.0 | 0.020497 | 13.62MiB/CPU-s | 206.5KiB/CPU-s | 2.66KiB/CPU-s | 0.0 | 0.014802 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 4.49KiB/CPU-s | 0.03 | 75.83% | 13.62MiB/CPU-s | 227.59KiB/CPU-s | 2.94KiB/CPU-s | 0.0 | 0.016316 | 13.63MiB/CPU-s | 191.17KiB/CPU-s | 2.47KiB/CPU-s | 0.0 | 0.0137 | False | False |
| splunk_hec_indexer_ack_blackhole | 2.22KiB/CPU-s | 0.02 | 37.88% | 13.62MiB/CPU-s | 250.2KiB/CPU-s | 3.23KiB/CPU-s | 0.0 | 0.017943 | 13.62MiB/CPU-s | 243.14KiB/CPU-s | 3.14KiB/CPU-s | 0.0 | 0.017434 | False | False |
| enterprise_http_to_http | 2.39KiB/CPU-s | 0.02 | 37.64% | 13.62MiB/CPU-s | 280.83KiB/CPU-s | 3.62KiB/CPU-s | 0.0 | 0.02014 | 13.62MiB/CPU-s | 252.79KiB/CPU-s | 3.26KiB/CPU-s | 0.0 | 0.018126 | False | False |
| http_to_http_noack | 1.86KiB/CPU-s | 0.01 | 23.31% | 13.61MiB/CPU-s | 354.97KiB/CPU-s | 4.58KiB/CPU-s | 0.0 | 0.025469 | 13.61MiB/CPU-s | 334.14KiB/CPU-s | 4.31KiB/CPU-s | 0.0 | 0.023971 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 1007.68B/CPU-s | 0.01 | 11.60% | 13.61MiB/CPU-s | 379.73KiB/CPU-s | 4.9KiB/CPU-s | 0.0 | 0.027234 | 13.62MiB/CPU-s | 359.57KiB/CPU-s | 4.64KiB/CPU-s | 0.0 | 0.025787 | False | False |
| file_to_blackhole | 64.71B/CPU-s | 0.0 | 0.24% | 54.5MiB/CPU-s | 1.11MiB/CPU-s | 14.62KiB/CPU-s | 0.0 | 0.020309 | 54.5MiB/CPU-s | 1.14MiB/CPU-s | 15.0KiB/CPU-s | 0.0 | 0.020857 | False | False |
| fluent_elasticsearch | -0B/CPU-s | -0.0 | 0.03% | 45.41MiB/CPU-s | 30.1KiB/CPU-s | 393.33B/CPU-s | 0.0 | 0.000647 | 45.41MiB/CPU-s | 30.34KiB/CPU-s | 396.46B/CPU-s | 0.0 | 0.000652 | False | False |
| datadog_agent_remap_datadog_logs | -24.44KiB/CPU-s | -0.07 | 53.68% | 32.74MiB/CPU-s | 2.01MiB/CPU-s | 26.5KiB/CPU-s | 0.0 | 0.061264 | 32.72MiB/CPU-s | 1.53MiB/CPU-s | 20.18KiB/CPU-s | 0.0 | 0.046695 | False | False |
| http_to_http_acks | -10.18KiB/CPU-s | -0.19 | 15.57% | 5.31MiB/CPU-s | 2.79MiB/CPU-s | 36.91KiB/CPU-s | 0.0 | 0.526061 | 5.3MiB/CPU-s | 2.76MiB/CPU-s | 36.41KiB/CPU-s | 0.0 | 0.519953 | True | False |
| otlp_http_to_blackhole | -7.63KiB/CPU-s | -0.49 | 99.96% | 1.53MiB/CPU-s | 113.81KiB/CPU-s | 1.47KiB/CPU-s | 0.0 | 0.072528 | 1.52MiB/CPU-s | 123.27KiB/CPU-s | 1.59KiB/CPU-s | 0.0 | 0.078943 | False | False |
| datadog_agent_remap_blackhole | -249.52KiB/CPU-s | -0.78 | 100.00% | 31.35MiB/CPU-s | 905.92KiB/CPU-s | 11.69KiB/CPU-s | 0.0 | 0.028219 | 31.1MiB/CPU-s | 868.53KiB/CPU-s | 11.21KiB/CPU-s | 0.0 | 0.027267 | False | False |
| otlp_grpc_to_blackhole | -17.28KiB/CPU-s | -1.63 | 100.00% | 1.03MiB/CPU-s | 39.64KiB/CPU-s | 524.04B/CPU-s | 0.0 | 0.037497 | 1.02MiB/CPU-s | 55.49KiB/CPU-s | 733.03B/CPU-s | 0.0 | 0.053359 | False | False |
| socket_to_socket_blackhole | -279.91KiB/CPU-s | -2.03 | 100.00% | 13.49MiB/CPU-s | 383.24KiB/CPU-s | 4.95KiB/CPU-s | 0.0 | 0.027741 | 13.22MiB/CPU-s | 341.43KiB/CPU-s | 4.41KiB/CPU-s | 0.0 | 0.025225 | False | False |
| syslog_regex_logs2metric_ddmetrics | -89.26KiB/CPU-s | -2.44 | 100.00% | 3.58MiB/CPU-s | 420.86KiB/CPU-s | 5.43KiB/CPU-s | 0.0 | 0.114859 | 3.49MiB/CPU-s | 457.61KiB/CPU-s | 5.91KiB/CPU-s | 0.0 | 0.128008 | True | False |
| datadog_agent_remap_datadog_logs_acks | -978.77KiB/CPU-s | -2.79 | 100.00% | 34.27MiB/CPU-s | 1.01MiB/CPU-s | 13.3KiB/CPU-s | 0.0 | 0.029374 | 33.31MiB/CPU-s | 1.18MiB/CPU-s | 15.65KiB/CPU-s | 0.0 | 0.035544 | False | False |
| http_text_to_http_json | -927.47KiB/CPU-s | -3.62 | 100.00% | 25.02MiB/CPU-s | 924.22KiB/CPU-s | 11.93KiB/CPU-s | 0.0 | 0.036065 | 24.12MiB/CPU-s | 541.37KiB/CPU-s | 6.98KiB/CPU-s | 0.0 | 0.021919 | False | False |
| syslog_humio_logs | -394.43KiB/CPU-s | -4.34 | 100.00% | 8.87MiB/CPU-s | 397.89KiB/CPU-s | 5.13KiB/CPU-s | 0.0 | 0.043824 | 8.48MiB/CPU-s | 174.88KiB/CPU-s | 2.26KiB/CPU-s | 0.0 | 0.020136 | False | False |
| syslog_log2metric_splunk_hec_metrics | -478.75KiB/CPU-s | -5.05 | 100.00% | 9.26MiB/CPU-s | 357.51KiB/CPU-s | 4.62KiB/CPU-s | 0.0 | 0.037685 | 8.8MiB/CPU-s | 443.88KiB/CPU-s | 5.73KiB/CPU-s | 0.0 | 0.049275 | False | False |
| syslog_splunk_hec_logs | -538.21KiB/CPU-s | -5.77 | 100.00% | 9.11MiB/CPU-s | 214.66KiB/CPU-s | 2.77KiB/CPU-s | 0.0 | 0.023013 | 8.58MiB/CPU-s | 302.85KiB/CPU-s | 3.91KiB/CPU-s | 0.0 | 0.034456 | False | False |
| syslog_loki | -573.25KiB/CPU-s | -6.33 | 100.00% | 8.85MiB/CPU-s | 322.29KiB/CPU-s | 4.16KiB/CPU-s | 0.0 | 0.03556 | 8.29MiB/CPU-s | 255.75KiB/CPU-s | 3.3KiB/CPU-s | 0.0 | 0.030123 | False | False |
Regression Test Results
Run ID: 12763a71-bb28-44f6-b2ea-a349d3bd77fd
Baseline: f90ca9e3e82b7577608b0c82a5feffb3f96d5f4c
Comparison: 02aba6bd56ce9b7f34756d3125ce7074383ac97d
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole | 1.15MiB/CPU-s | 3.77 | 100.00% | 30.48MiB/CPU-s | 2.04MiB/CPU-s | 26.95KiB/CPU-s | 0.0 | 0.066866 | 31.62MiB/CPU-s | 577.95KiB/CPU-s | 7.46KiB/CPU-s | 0.0 | 0.017846 | False | False |
| datadog_agent_remap_datadog_logs | 491.93KiB/CPU-s | 1.43 | 100.00% | 33.69MiB/CPU-s | 1.44MiB/CPU-s | 19.02KiB/CPU-s | 0.0 | 0.042728 | 34.17MiB/CPU-s | 1.19MiB/CPU-s | 15.7KiB/CPU-s | 0.0 | 0.03478 | False | False |
| splunk_hec_route_s3 | 126.04KiB/CPU-s | 1.07 | 100.00% | 11.54MiB/CPU-s | 605.16KiB/CPU-s | 7.81KiB/CPU-s | 0.0 | 0.05122 | 11.66MiB/CPU-s | 538.78KiB/CPU-s | 6.95KiB/CPU-s | 0.0 | 0.04512 | False | False |
| otlp_http_to_blackhole | 6.94KiB/CPU-s | 0.44 | 99.94% | 1.54MiB/CPU-s | 109.88KiB/CPU-s | 1.42KiB/CPU-s | 0.0 | 0.069458 | 1.55MiB/CPU-s | 110.87KiB/CPU-s | 1.43KiB/CPU-s | 0.0 | 0.069779 | False | False |
| syslog_humio_logs | 32.29KiB/CPU-s | 0.35 | 100.00% | 9.14MiB/CPU-s | 284.18KiB/CPU-s | 3.67KiB/CPU-s | 0.0 | 0.030376 | 9.17MiB/CPU-s | 264.07KiB/CPU-s | 3.41KiB/CPU-s | 0.0 | 0.028129 | False | False |
| syslog_regex_logs2metric_ddmetrics | 8.59KiB/CPU-s | 0.23 | 72.66% | 3.64MiB/CPU-s | 446.95KiB/CPU-s | 5.77KiB/CPU-s | 0.0 | 0.119859 | 3.65MiB/CPU-s | 411.78KiB/CPU-s | 5.32KiB/CPU-s | 0.0 | 0.110172 | True | False |
| http_to_http_noack | 4.16KiB/CPU-s | 0.03 | 57.48% | 13.61MiB/CPU-s | 311.03KiB/CPU-s | 4.01KiB/CPU-s | 0.0 | 0.022309 | 13.62MiB/CPU-s | 258.35KiB/CPU-s | 3.33KiB/CPU-s | 0.0 | 0.018525 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 3.47KiB/CPU-s | 0.02 | 64.89% | 13.62MiB/CPU-s | 215.28KiB/CPU-s | 2.78KiB/CPU-s | 0.0 | 0.015432 | 13.63MiB/CPU-s | 191.38KiB/CPU-s | 2.47KiB/CPU-s | 0.0 | 0.013716 | False | False |
| splunk_hec_indexer_ack_blackhole | 1.58KiB/CPU-s | 0.01 | 27.79% | 13.62MiB/CPU-s | 248.31KiB/CPU-s | 3.2KiB/CPU-s | 0.0 | 0.017806 | 13.62MiB/CPU-s | 239.18KiB/CPU-s | 3.09KiB/CPU-s | 0.0 | 0.01715 | False | False |
| enterprise_http_to_http | -68.59B/CPU-s | -0.0 | 1.96% | 13.62MiB/CPU-s | 150.12KiB/CPU-s | 1.94KiB/CPU-s | 0.0 | 0.01076 | 13.62MiB/CPU-s | 148.87KiB/CPU-s | 1.92KiB/CPU-s | 0.0 | 0.01067 | False | False |
| fluent_elasticsearch | 61.13B/CPU-s | 0.0 | 8.79% | 45.41MiB/CPU-s | 29.81KiB/CPU-s | 389.7B/CPU-s | 0.0 | 0.000641 | 45.41MiB/CPU-s | 30.08KiB/CPU-s | 393.18B/CPU-s | 0.0 | 0.000647 | False | False |
| file_to_blackhole | -3.49KiB/CPU-s | -0.01 | 12.69% | 54.5MiB/CPU-s | 1.18MiB/CPU-s | 15.64KiB/CPU-s | 0.0 | 0.021722 | 54.49MiB/CPU-s | 1.15MiB/CPU-s | 15.23KiB/CPU-s | 0.0 | 0.021183 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -869.65B/CPU-s | -0.01 | 10.63% | 13.62MiB/CPU-s | 341.43KiB/CPU-s | 4.41KiB/CPU-s | 0.0 | 0.024485 | 13.62MiB/CPU-s | 354.96KiB/CPU-s | 4.58KiB/CPU-s | 0.0 | 0.025456 | False | False |
| syslog_loki | -9.29KiB/CPU-s | -0.1 | 98.84% | 8.78MiB/CPU-s | 176.89KiB/CPU-s | 2.28KiB/CPU-s | 0.0 | 0.019675 | 8.77MiB/CPU-s | 223.67KiB/CPU-s | 2.89KiB/CPU-s | 0.0 | 0.024905 | False | False |
| datadog_agent_remap_datadog_logs_acks | -118.04KiB/CPU-s | -0.34 | 100.00% | 33.9MiB/CPU-s | 942.92KiB/CPU-s | 12.17KiB/CPU-s | 0.0 | 0.027161 | 33.78MiB/CPU-s | 775.05KiB/CPU-s | 10.01KiB/CPU-s | 0.0 | 0.022402 | False | False |
| otlp_grpc_to_blackhole | -5.27KiB/CPU-s | -0.49 | 100.00% | 1.05MiB/CPU-s | 45.73KiB/CPU-s | 604.39B/CPU-s | 0.0 | 0.04237 | 1.05MiB/CPU-s | 55.04KiB/CPU-s | 727.26B/CPU-s | 0.0 | 0.051246 | False | False |
| http_text_to_http_json | -143.76KiB/CPU-s | -0.59 | 100.00% | 23.63MiB/CPU-s | 604.69KiB/CPU-s | 7.81KiB/CPU-s | 0.0 | 0.024988 | 23.49MiB/CPU-s | 754.71KiB/CPU-s | 9.74KiB/CPU-s | 0.0 | 0.031374 | False | False |
| syslog_log2metric_splunk_hec_metrics | -75.15KiB/CPU-s | -0.77 | 100.00% | 9.47MiB/CPU-s | 306.34KiB/CPU-s | 3.95KiB/CPU-s | 0.0 | 0.031585 | 9.4MiB/CPU-s | 294.97KiB/CPU-s | 3.81KiB/CPU-s | 0.0 | 0.03065 | False | False |
| http_to_http_acks | -59.78KiB/CPU-s | -1.1 | 74.98% | 5.29MiB/CPU-s | 2.82MiB/CPU-s | 37.27KiB/CPU-s | 0.0 | 0.5329 | 5.23MiB/CPU-s | 2.74MiB/CPU-s | 36.25KiB/CPU-s | 0.0 | 0.524091 | True | False |
| http_to_http_json | -167.03KiB/CPU-s | -1.21 | 100.00% | 13.52MiB/CPU-s | 345.77KiB/CPU-s | 4.46KiB/CPU-s | 0.0 | 0.024979 | 13.35MiB/CPU-s | 551.3KiB/CPU-s | 7.11KiB/CPU-s | 0.0 | 0.040313 | False | False |
| socket_to_socket_blackhole | -172.43KiB/CPU-s | -1.25 | 100.00% | 13.5MiB/CPU-s | 204.12KiB/CPU-s | 2.63KiB/CPU-s | 0.0 | 0.014766 | 13.33MiB/CPU-s | 321.91KiB/CPU-s | 4.16KiB/CPU-s | 0.0 | 0.023581 | False | False |
| syslog_log2metric_humio_metrics | -129.63KiB/CPU-s | -2.1 | 100.00% | 6.04MiB/CPU-s | 334.63KiB/CPU-s | 4.32KiB/CPU-s | 0.0 | 0.054138 | 5.91MiB/CPU-s | 441.08KiB/CPU-s | 5.69KiB/CPU-s | 0.0 | 0.07289 | False | False |
| datadog_agent_remap_blackhole_acks | -758.65KiB/CPU-s | -2.32 | 100.00% | 31.99MiB/CPU-s | 1.33MiB/CPU-s | 17.61KiB/CPU-s | 0.0 | 0.041623 | 31.25MiB/CPU-s | 1.35MiB/CPU-s | 17.9KiB/CPU-s | 0.0 | 0.043314 | False | False |
| syslog_splunk_hec_logs | -244.7KiB/CPU-s | -2.63 | 100.00% | 9.1MiB/CPU-s | 187.63KiB/CPU-s | 2.42KiB/CPU-s | 0.0 | 0.020136 | 8.86MiB/CPU-s | 338.19KiB/CPU-s | 4.36KiB/CPU-s | 0.0 | 0.037271 | False | False |
Regression Test Results
Run ID: b023387e-cfdc-4d1a-9991-458ddc3ccc73
Baseline: a1b87b51c68cbff5e5c12672753e1780a5e50f6a
Comparison: f63bf6da9b043819c4540b043c22e0b5bcab1b91
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole_acks | 1.28MiB/CPU-s | 4.34 | 100.00% | 29.45MiB/CPU-s | 6.82MiB/CPU-s | 90.13KiB/CPU-s | 0.0 | 0.231405 | 30.73MiB/CPU-s | 1.81MiB/CPU-s | 23.9KiB/CPU-s | 0.0 | 0.058808 | True | False |
| syslog_regex_logs2metric_ddmetrics | 70.09KiB/CPU-s | 1.84 | 100.00% | 3.73MiB/CPU-s | 470.42KiB/CPU-s | 6.07KiB/CPU-s | 0.0 | 0.123171 | 3.8MiB/CPU-s | 456.37KiB/CPU-s | 5.89KiB/CPU-s | 0.0 | 0.117339 | True | False |
| datadog_agent_remap_blackhole | 439.4KiB/CPU-s | 1.41 | 100.00% | 30.35MiB/CPU-s | 1.98MiB/CPU-s | 26.18KiB/CPU-s | 0.0 | 0.065238 | 30.78MiB/CPU-s | 1.96MiB/CPU-s | 25.87KiB/CPU-s | 0.0 | 0.063582 | False | False |
| socket_to_socket_blackhole | 172.54KiB/CPU-s | 1.26 | 100.00% | 13.33MiB/CPU-s | 133.32KiB/CPU-s | 1.72KiB/CPU-s | 0.0 | 0.009765 | 13.5MiB/CPU-s | 94.47KiB/CPU-s | 1.22KiB/CPU-s | 0.0 | 0.006833 | False | False |
| syslog_loki | 105.95KiB/CPU-s | 1.22 | 100.00% | 8.48MiB/CPU-s | 433.37KiB/CPU-s | 5.6KiB/CPU-s | 0.0 | 0.049891 | 8.59MiB/CPU-s | 379.99KiB/CPU-s | 4.91KiB/CPU-s | 0.0 | 0.043218 | False | False |
| otlp_grpc_to_blackhole | 10.3KiB/CPU-s | 0.97 | 100.00% | 1.04MiB/CPU-s | 40.96KiB/CPU-s | 541.47B/CPU-s | 0.0 | 0.038476 | 1.05MiB/CPU-s | 55.3KiB/CPU-s | 730.64B/CPU-s | 0.0 | 0.051445 | False | False |
| syslog_log2metric_humio_metrics | 47.92KiB/CPU-s | 0.77 | 100.00% | 6.07MiB/CPU-s | 378.98KiB/CPU-s | 4.89KiB/CPU-s | 0.0 | 0.06099 | 6.11MiB/CPU-s | 243.46KiB/CPU-s | 3.14KiB/CPU-s | 0.0 | 0.038881 | False | False |
| otlp_http_to_blackhole | 10.25KiB/CPU-s | 0.65 | 100.00% | 1.53MiB/CPU-s | 110.28KiB/CPU-s | 1.42KiB/CPU-s | 0.0 | 0.070343 | 1.54MiB/CPU-s | 123.58KiB/CPU-s | 1.59KiB/CPU-s | 0.0 | 0.078315 | False | False |
| splunk_hec_route_s3 | 74.12KiB/CPU-s | 0.63 | 100.00% | 11.48MiB/CPU-s | 731.13KiB/CPU-s | 9.43KiB/CPU-s | 0.0 | 0.062163 | 11.56MiB/CPU-s | 697.47KiB/CPU-s | 9.0KiB/CPU-s | 0.0 | 0.05893 | False | False |
| syslog_humio_logs | 9.21KiB/CPU-s | 0.1 | 99.99% | 9.24MiB/CPU-s | 159.55KiB/CPU-s | 2.06KiB/CPU-s | 0.0 | 0.01686 | 9.25MiB/CPU-s | 79.23KiB/CPU-s | 1.02KiB/CPU-s | 0.0 | 0.008364 | False | False |
| http_to_http_noack | 4.12KiB/CPU-s | 0.03 | 50.89% | 13.61MiB/CPU-s | 349.3KiB/CPU-s | 4.51KiB/CPU-s | 0.0 | 0.025062 | 13.61MiB/CPU-s | 305.55KiB/CPU-s | 3.94KiB/CPU-s | 0.0 | 0.021916 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 4.74KiB/CPU-s | 0.03 | 78.19% | 13.62MiB/CPU-s | 227.64KiB/CPU-s | 2.94KiB/CPU-s | 0.0 | 0.01632 | 13.63MiB/CPU-s | 192.16KiB/CPU-s | 2.48KiB/CPU-s | 0.0 | 0.013772 | False | False |
| fluent_elasticsearch | -124.5B/CPU-s | -0.0 | 17.73% | 45.41MiB/CPU-s | 30.03KiB/CPU-s | 392.55B/CPU-s | 0.0 | 0.000646 | 45.41MiB/CPU-s | 30.06KiB/CPU-s | 393.12B/CPU-s | 0.0 | 0.000646 | False | False |
| splunk_hec_indexer_ack_blackhole | 153.62B/CPU-s | 0.0 | 2.61% | 13.62MiB/CPU-s | 253.65KiB/CPU-s | 3.27KiB/CPU-s | 0.0 | 0.01819 | 13.62MiB/CPU-s | 249.65KiB/CPU-s | 3.22KiB/CPU-s | 0.0 | 0.017902 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -258.72B/CPU-s | -0.0 | 3.24% | 13.62MiB/CPU-s | 340.45KiB/CPU-s | 4.39KiB/CPU-s | 0.0 | 0.024414 | 13.62MiB/CPU-s | 340.6KiB/CPU-s | 4.39KiB/CPU-s | 0.0 | 0.024426 | False | False |
| file_to_blackhole | -3.28KiB/CPU-s | -0.01 | 12.26% | 54.49MiB/CPU-s | 1.09MiB/CPU-s | 14.46KiB/CPU-s | 0.0 | 0.020083 | 54.49MiB/CPU-s | 1.18MiB/CPU-s | 15.59KiB/CPU-s | 0.0 | 0.021678 | False | False |
| enterprise_http_to_http | -2.1KiB/CPU-s | -0.02 | 36.92% | 13.62MiB/CPU-s | 222.9KiB/CPU-s | 2.88KiB/CPU-s | 0.0 | 0.015981 | 13.62MiB/CPU-s | 255.43KiB/CPU-s | 3.3KiB/CPU-s | 0.0 | 0.018315 | False | False |
| syslog_splunk_hec_logs | -41.85KiB/CPU-s | -0.45 | 100.00% | 9.16MiB/CPU-s | 100.21KiB/CPU-s | 1.29KiB/CPU-s | 0.0 | 0.010684 | 9.12MiB/CPU-s | 185.63KiB/CPU-s | 2.39KiB/CPU-s | 0.0 | 0.019878 | False | False |
| http_to_http_acks | -32.53KiB/CPU-s | -0.61 | 47.03% | 5.23MiB/CPU-s | 2.75MiB/CPU-s | 36.3KiB/CPU-s | 0.0 | 0.525414 | 5.2MiB/CPU-s | 2.79MiB/CPU-s | 36.91KiB/CPU-s | 0.0 | 0.537192 | True | False |
| http_to_http_json | -107.01KiB/CPU-s | -0.77 | 100.00% | 13.62MiB/CPU-s | 209.47KiB/CPU-s | 2.7KiB/CPU-s | 0.0 | 0.015015 | 13.52MiB/CPU-s | 375.39KiB/CPU-s | 4.84KiB/CPU-s | 0.0 | 0.027116 | False | False |
| syslog_log2metric_splunk_hec_metrics | -105.87KiB/CPU-s | -1.09 | 100.00% | 9.5MiB/CPU-s | 207.69KiB/CPU-s | 2.68KiB/CPU-s | 0.0 | 0.021354 | 9.39MiB/CPU-s | 186.29KiB/CPU-s | 2.41KiB/CPU-s | 0.0 | 0.019366 | False | False |
| datadog_agent_remap_datadog_logs_acks | -798.91KiB/CPU-s | -2.31 | 100.00% | 33.78MiB/CPU-s | 1.22MiB/CPU-s | 16.08KiB/CPU-s | 0.0 | 0.03603 | 33.0MiB/CPU-s | 1.75MiB/CPU-s | 23.06KiB/CPU-s | 0.0 | 0.052877 | False | False |
| http_text_to_http_json | -645.26KiB/CPU-s | -2.64 | 100.00% | 23.89MiB/CPU-s | 1.0MiB/CPU-s | 13.25KiB/CPU-s | 0.0 | 0.041951 | 23.26MiB/CPU-s | 874.15KiB/CPU-s | 11.28KiB/CPU-s | 0.0 | 0.036691 | False | False |
| datadog_agent_remap_datadog_logs | -934.62KiB/CPU-s | -2.65 | 100.00% | 34.48MiB/CPU-s | 1.2MiB/CPU-s | 15.87KiB/CPU-s | 0.0 | 0.034834 | 33.57MiB/CPU-s | 1.71MiB/CPU-s | 22.52KiB/CPU-s | 0.0 | 0.050792 | False | False |