vector
vector copied to clipboard
vectordotdev
chore(deps): bump serde_with from 2.0.0 to 2.0.1
Bumps serde_with from 2.0.0 to 2.0.1.
Release notes
Sourced from serde_with's releases.
serde_with_macros v2.0.1
Changed
- Warn if
serde_asis used on an enum variant. Attributes on enum variants were never supported. But#[serde(with = "...")]can be added on variants, such that some confusion can occur when migration (#499).serde_with v2.0.1
Added
timeadded support for the well-knownIso8601format. This extends the existing support ofRfc2822andRfc3339.Changed
- Warn if
serde_asis used on an enum variant. Attributes on enum variants were never supported. But#[serde(with = "...")]can be added on variants, such that some confusion can occur when migration (#499).Note
A cargo bug (cargo#10801) means that upgrading from v1 to v2 may add unnecessary crates to the
Cargo.lockfile. A diff of the lock-file makes it seem thatserde_withdepends on new crates, even though these crates are unused and will not get compiled or linked. However, tools consumingCargo.lockorcargo metadatamight give wrong results.
Commits
7753240Merge #515cb8efccBump versions to v2.0.17718cdcMerge #514cad71a3Do not put the badges into the headers850b89aReplace binder link with rustexplorer9454ca7Merge #5137abb244Gate serde_with tests on the correct featurese4961b3Fix warningae34880Rework proc-macro generated code to avoid::stdpaths97f9aeaMerge #512- Additional commits viewable in compare view
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.
Deploy Preview for vector-project canceled.
| Name | Link |
|---|---|
| Latest commit | 8cadae92b8a0bfb7181510cc38ebd1aca943b333 |
| Latest deploy log | https://app.netlify.com/sites/vector-project/deploys/6331a4dd697d320008df3add |
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
Soak Test Results
Baseline: 0083d15aa7ba31da598c63a05ba8d034edb976fa Comparison: 172b0a4045141604a46c7568759ade343903b06c Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| socket_to_socket_blackhole | 381.52KiB | 1.65 | 100.00% | 22.56MiB | 176.73KiB | 3.61KiB | 0 | 0.00764738 | 22.94MiB | 124.49KiB | 2.54KiB | 0 | 0.00529909 | False | False |
| syslog_loki | 209.04KiB | 1.42 | 100.00% | 14.38MiB | 295.76KiB | 6.06KiB | 0 | 0.0200866 | 14.58MiB | 740.26KiB | 15.05KiB | 0 | 0.0495704 | False | False |
| splunk_hec_route_s3 | 262.45KiB | 1.39 | 99.99% | 18.45MiB | 2.31MiB | 48.06KiB | 0 | 0.125045 | 18.71MiB | 2.21MiB | 46.18KiB | 0 | 0.117993 | False | False |
| datadog_agent_remap_blackhole_acks | 770.61KiB | 1.22 | 100.00% | 61.65MiB | 3.94MiB | 82.04KiB | 0 | 0.0638573 | 62.41MiB | 2.06MiB | 43.09KiB | 0 | 0.032954 | False | False |
| datadog_agent_remap_blackhole | 447.16KiB | 0.75 | 99.99% | 58.22MiB | 4.35MiB | 90.65KiB | 0 | 0.0747237 | 58.66MiB | 3.53MiB | 73.61KiB | 0 | 0.0601157 | False | False |
| http_pipelines_blackhole_acks | 9.17KiB | 0.73 | 99.96% | 1.22MiB | 112.89KiB | 2.3KiB | 0 | 0.0904414 | 1.23MiB | 57.68KiB | 1.18KiB | 0 | 0.0458771 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 15.91KiB | 0.07 | 82.00% | 23.82MiB | 476.34KiB | 9.72KiB | 0 | 0.0195221 | 23.84MiB | 333.26KiB | 6.8KiB | 0 | 0.0136494 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 4.46KiB | 0.02 | 14.51% | 23.75MiB | 852.71KiB | 17.34KiB | 0 | 0.0350558 | 23.75MiB | 843.95KiB | 17.17KiB | 0 | 0.0346895 | False | False |
| enterprise_http_to_http | 4.39KiB | 0.02 | 44.77% | 23.84MiB | 254.45KiB | 5.19KiB | 0 | 0.01042 | 23.85MiB | 256.29KiB | 5.24KiB | 0 | 0.0104934 | False | False |
| splunk_hec_indexer_ack_blackhole | 1.98KiB | 0.01 | 6.17% | 23.75MiB | 898.23KiB | 18.27KiB | 0 | 0.0369227 | 23.75MiB | 878.22KiB | 17.87KiB | 0 | 0.0360971 | False | False |
| file_to_blackhole | -39.82KiB | -0.04 | 27.53% | 95.33MiB | 3.76MiB | 77.94KiB | 0 | 0.0394294 | 95.3MiB | 3.94MiB | 81.92KiB | 0 | 0.0413067 | False | False |
| syslog_regex_logs2metric_ddmetrics | -7.01KiB | -0.05 | 33.42% | 12.64MiB | 612.07KiB | 12.47KiB | 0 | 0.0472723 | 12.63MiB | 508.61KiB | 10.37KiB | 0 | 0.0393028 | False | False |
| http_to_http_json | -40.65KiB | -0.17 | 99.80% | 23.85MiB | 347.14KiB | 7.09KiB | 0 | 0.0142131 | 23.81MiB | 542.74KiB | 11.07KiB | 0 | 0.0222587 | False | False |
| syslog_splunk_hec_logs | -29.82KiB | -0.18 | 87.85% | 16.16MiB | 730.5KiB | 14.87KiB | 0 | 0.0441265 | 16.13MiB | 599.59KiB | 12.23KiB | 0 | 0.0362839 | False | False |
| http_to_http_acks | -37.68KiB | -0.21 | 12.55% | 17.41MiB | 8.01MiB | 167.5KiB | 0 | 0.460272 | 17.37MiB | 8.13MiB | 169.9KiB | 0 | 0.467836 | True | True |
| syslog_log2metric_splunk_hec_metrics | -48.26KiB | -0.26 | 98.74% | 17.84MiB | 598.68KiB | 12.2KiB | 0 | 0.0327618 | 17.79MiB | 736.67KiB | 14.99KiB | 0 | 0.0404199 | False | False |
| fluent_elasticsearch | -210.41KiB | -0.26 | 100.00% | 79.47MiB | 52.9KiB | 1.07KiB | 0 | 0.000649868 | 79.27MiB | 1.71MiB | 35.09KiB | 0 | 0.0215241 | False | False |
| syslog_log2metric_humio_metrics | -66.58KiB | -0.51 | 100.00% | 12.83MiB | 187.84KiB | 3.84KiB | 0 | 0.0142986 | 12.76MiB | 589.17KiB | 11.98KiB | 0 | 0.0450768 | False | False |
| http_to_http_noack | -131.12KiB | -0.54 | 100.00% | 23.85MiB | 255.8KiB | 5.23KiB | 0 | 0.010474 | 23.72MiB | 1.23MiB | 25.7KiB | 0 | 0.0519877 | False | False |
| syslog_humio_logs | -123.92KiB | -0.73 | 100.00% | 16.54MiB | 162.43KiB | 3.32KiB | 0 | 0.00959015 | 16.42MiB | 179.21KiB | 3.67KiB | 0 | 0.0106586 | False | False |
| http_pipelines_blackhole | -13.2KiB | -0.76 | 100.00% | 1.69MiB | 9.54KiB | 199.65B | 0 | 0.00550178 | 1.68MiB | 140.8KiB | 2.87KiB | 0 | 0.0817966 | False | False |
| http_text_to_http_json | -299.57KiB | -0.78 | 100.00% | 37.52MiB | 1.21MiB | 25.37KiB | 0 | 0.0323509 | 37.22MiB | 1.05MiB | 21.92KiB | 0 | 0.0281495 | False | False |
| datadog_agent_remap_datadog_logs | -581.04KiB | -0.91 | 100.00% | 62.46MiB | 231.68KiB | 4.74KiB | 0 | 0.00362131 | 61.9MiB | 4.17MiB | 86.87KiB | 0 | 0.0673784 | False | False |
| datadog_agent_remap_datadog_logs_acks | -663.07KiB | -1.06 | 100.00% | 61.05MiB | 2.39MiB | 50.11KiB | 0 | 0.0391832 | 60.4MiB | 4.37MiB | 90.95KiB | 0 | 0.072325 | False | False |
| http_pipelines_no_grok_blackhole | -182.62KiB | -1.64 | 100.00% | 10.87MiB | 108.23KiB | 2.21KiB | 0 | 0.00972386 | 10.69MiB | 1.07MiB | 22.18KiB | 0 | 0.0996628 | False | False |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Soak Test Results
Baseline: 96003531105213c8e2a245f68605e6764c261850 Comparison: bb9aa29caea77a3205b945e38afe626a29261528 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| socket_to_socket_blackhole | 811.97KiB | 3.4 | 100.00% | 23.33MiB | 144.28KiB | 2.95KiB | 0 | 0.00603849 | 24.12MiB | 213.55KiB | 4.36KiB | 0 | 0.00864361 | False | False |
| http_to_http_acks | 568.5KiB | 3.23 | 98.64% | 17.17MiB | 7.84MiB | 163.8KiB | 0 | 0.456191 | 17.73MiB | 7.77MiB | 162.0KiB | 0 | 0.437871 | True | True |
| splunk_hec_route_s3 | 582.99KiB | 3.07 | 100.00% | 18.56MiB | 2.34MiB | 48.78KiB | 0 | 0.126169 | 19.13MiB | 2.15MiB | 45.03KiB | 0 | 0.112389 | False | False |
| http_text_to_http_json | 1.1MiB | 2.85 | 100.00% | 38.57MiB | 986.8KiB | 20.14KiB | 0 | 0.0249808 | 39.67MiB | 783.21KiB | 15.99KiB | 0 | 0.019278 | False | False |
| datadog_agent_remap_datadog_logs | 1022.75KiB | 1.76 | 100.00% | 56.9MiB | 4.21MiB | 88.15KiB | 0 | 0.0739163 | 57.9MiB | 5.76MiB | 119.93KiB | 0 | 0.0993832 | False | False |
| http_pipelines_blackhole_acks | 2.99KiB | 0.27 | 47.74% | 1.1MiB | 169.51KiB | 3.45KiB | 0 | 0.150235 | 1.1MiB | 154.74KiB | 3.16KiB | 0 | 0.136781 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 16.01KiB | 0.07 | 82.38% | 23.82MiB | 473.04KiB | 9.66KiB | 0 | 0.0193882 | 23.84MiB | 334.72KiB | 6.84KiB | 0 | 0.0137098 | False | False |
| enterprise_http_to_http | -908.41B | -0 | 9.72% | 23.85MiB | 253.39KiB | 5.17KiB | 0 | 0.0103747 | 23.85MiB | 249.45KiB | 5.1KiB | 0 | 0.0102137 | False | False |
| splunk_hec_indexer_ack_blackhole | -5.6KiB | -0.02 | 17.08% | 23.75MiB | 893.4KiB | 18.17KiB | 0 | 0.0367282 | 23.74MiB | 910.66KiB | 18.52KiB | 0 | 0.0374464 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -14.35KiB | -0.06 | 45.18% | 23.77MiB | 824.27KiB | 16.78KiB | 0 | 0.0338593 | 23.75MiB | 836.52KiB | 17.02KiB | 0 | 0.0343829 | False | False |
| file_to_blackhole | -65.51KiB | -0.07 | 41.75% | 95.33MiB | 3.58MiB | 74.21KiB | 0 | 0.0375621 | 95.26MiB | 4.48MiB | 93.22KiB | 0 | 0.047071 | False | False |
| syslog_log2metric_humio_metrics | -23.87KiB | -0.18 | 95.68% | 13.02MiB | 240.43KiB | 4.91KiB | 0 | 0.0180334 | 12.99MiB | 527.36KiB | 10.74KiB | 0 | 0.0396254 | False | False |
| fluent_elasticsearch | -199.52KiB | -0.25 | 100.00% | 79.47MiB | 53.81KiB | 1.09KiB | 0 | 0.000661046 | 79.28MiB | 1.81MiB | 37.36KiB | 0 | 0.0228863 | False | False |
| http_to_http_noack | -86.88KiB | -0.36 | 99.98% | 23.84MiB | 406.96KiB | 8.32KiB | 0 | 0.0166686 | 23.75MiB | 1.05MiB | 22.0KiB | 0 | 0.0444033 | False | False |
| datadog_agent_remap_blackhole_acks | -303.44KiB | -0.51 | 98.31% | 58.61MiB | 4.88MiB | 101.61KiB | 0 | 0.0832014 | 58.31MiB | 3.64MiB | 76.13KiB | 0 | 0.0623708 | False | False |
| http_to_http_json | -154.77KiB | -0.63 | 100.00% | 23.85MiB | 337.52KiB | 6.89KiB | 0 | 0.0138192 | 23.7MiB | 760.86KiB | 15.54KiB | 0 | 0.0313508 | False | False |
| datadog_agent_remap_blackhole | -566.18KiB | -0.9 | 100.00% | 61.13MiB | 3.84MiB | 80.03KiB | 0 | 0.0627874 | 60.58MiB | 3.31MiB | 69.06KiB | 0 | 0.054651 | False | False |
| syslog_regex_logs2metric_ddmetrics | -168.68KiB | -1.31 | 100.00% | 12.58MiB | 615.35KiB | 12.53KiB | 0 | 0.0477474 | 12.42MiB | 486.05KiB | 9.91KiB | 0 | 0.0382151 | False | False |
| syslog_loki | -198.18KiB | -1.32 | 100.00% | 14.69MiB | 416.14KiB | 8.51KiB | 0 | 0.0276643 | 14.49MiB | 729.12KiB | 14.82KiB | 0 | 0.0491182 | False | False |
| syslog_splunk_hec_logs | -224.0KiB | -1.35 | 100.00% | 16.25MiB | 828.29KiB | 16.84KiB | 0 | 0.0497563 | 16.03MiB | 627.85KiB | 12.81KiB | 0 | 0.0382304 | False | False |
| datadog_agent_remap_datadog_logs_acks | -848.84KiB | -1.35 | 100.00% | 61.28MiB | 3.37MiB | 70.53KiB | 0 | 0.0549806 | 60.46MiB | 5.24MiB | 109.12KiB | 0 | 0.0866908 | False | False |
| http_pipelines_no_grok_blackhole | -148.78KiB | -1.4 | 100.00% | 10.41MiB | 499.83KiB | 10.2KiB | 0 | 0.0468758 | 10.27MiB | 1.07MiB | 22.34KiB | 0 | 0.104478 | False | False |
| syslog_humio_logs | -250.03KiB | -1.49 | 100.00% | 16.44MiB | 120.31KiB | 2.46KiB | 0 | 0.00714417 | 16.2MiB | 147.41KiB | 3.02KiB | 0 | 0.00888576 | False | False |
| http_pipelines_blackhole | -25.48KiB | -1.51 | 100.00% | 1.65MiB | 28.17KiB | 589.08B | 0 | 0.016678 | 1.62MiB | 95.43KiB | 1.95KiB | 0 | 0.057361 | False | False |
| syslog_log2metric_splunk_hec_metrics | -303.68KiB | -1.72 | 100.00% | 17.23MiB | 920.86KiB | 18.76KiB | 0 | 0.0521852 | 16.93MiB | 1008.45KiB | 20.55KiB | 0 | 0.0581496 | False | False |
Soak Test Results
Baseline: 5d1550a2056f32a885f4d13df3a75b45f777440a Comparison: 1cdf5ca39fafcfa1378e9c51af6a579e94f4108f Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_text_to_http_json | 1.04MiB | 2.68 | 100.00% | 38.65MiB | 974.97KiB | 19.9KiB | 0 | 0.0246271 | 39.69MiB | 1.0MiB | 20.98KiB | 0 | 0.0252731 | False | False |
| syslog_log2metric_humio_metrics | 267.11KiB | 2.11 | 100.00% | 12.38MiB | 295.04KiB | 6.03KiB | 0 | 0.0232781 | 12.64MiB | 537.66KiB | 10.95KiB | 0 | 0.0415447 | False | False |
| syslog_loki | 233.83KiB | 1.6 | 100.00% | 14.29MiB | 456.99KiB | 9.35KiB | 0 | 0.0312153 | 14.52MiB | 752.75KiB | 15.3KiB | 0 | 0.0506087 | False | False |
| datadog_agent_remap_blackhole | 915.97KiB | 1.5 | 100.00% | 59.75MiB | 3.99MiB | 83.2KiB | 0 | 0.0668132 | 60.64MiB | 2.27MiB | 47.37KiB | 0 | 0.0374025 | False | False |
| splunk_hec_route_s3 | 281.26KiB | 1.47 | 100.00% | 18.74MiB | 2.33MiB | 48.51KiB | 0 | 0.124298 | 19.02MiB | 2.17MiB | 45.46KiB | 0 | 0.11435 | False | False |
| socket_to_socket_blackhole | 347.09KiB | 1.43 | 100.00% | 23.67MiB | 239.0KiB | 4.88KiB | 0 | 0.00985666 | 24.01MiB | 149.01KiB | 3.04KiB | 0 | 0.00605855 | False | False |
| datadog_agent_remap_blackhole_acks | 663.18KiB | 1.08 | 100.00% | 60.14MiB | 4.51MiB | 93.89KiB | 0 | 0.0749613 | 60.79MiB | 2.84MiB | 59.34KiB | 0 | 0.0466479 | False | False |
| syslog_splunk_hec_logs | 147.68KiB | 0.88 | 100.00% | 16.31MiB | 769.98KiB | 15.66KiB | 0 | 0.0460906 | 16.46MiB | 528.89KiB | 10.8KiB | 0 | 0.0313816 | False | False |
| http_pipelines_blackhole_acks | 9.91KiB | 0.79 | 99.98% | 1.23MiB | 103.37KiB | 2.1KiB | 0 | 0.0819916 | 1.24MiB | 80.13KiB | 1.63KiB | 0 | 0.0630615 | False | False |
| syslog_log2metric_splunk_hec_metrics | 90.42KiB | 0.51 | 100.00% | 17.25MiB | 550.54KiB | 11.22KiB | 0 | 0.031162 | 17.34MiB | 855.14KiB | 17.39KiB | 0 | 0.0481564 | False | False |
| http_to_http_acks | 45.28KiB | 0.26 | 15.20% | 17.19MiB | 8.01MiB | 167.37KiB | 0 | 0.465749 | 17.23MiB | 7.99MiB | 166.75KiB | 0 | 0.463502 | True | True |
| splunk_hec_indexer_ack_blackhole | 22.44KiB | 0.09 | 61.26% | 23.73MiB | 937.36KiB | 19.05KiB | 0 | 0.038561 | 23.76MiB | 866.48KiB | 17.63KiB | 0 | 0.0356126 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 17.68KiB | 0.07 | 51.84% | 23.74MiB | 917.17KiB | 18.65KiB | 0 | 0.0377165 | 23.76MiB | 827.76KiB | 16.84KiB | 0 | 0.0340148 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 8.8KiB | 0.04 | 57.14% | 23.83MiB | 434.02KiB | 8.86KiB | 0 | 0.0177843 | 23.84MiB | 329.08KiB | 6.72KiB | 0 | 0.0134796 | False | False |
| enterprise_http_to_http | -2.11KiB | -0.01 | 22.81% | 23.85MiB | 250.21KiB | 5.11KiB | 0 | 0.0102443 | 23.84MiB | 254.3KiB | 5.21KiB | 0 | 0.0104128 | False | False |
| file_to_blackhole | -66.34KiB | -0.07 | 47.87% | 95.34MiB | 3.31MiB | 68.67KiB | 0 | 0.0347391 | 95.27MiB | 3.72MiB | 77.36KiB | 0 | 0.0390723 | False | False |
| syslog_humio_logs | -16.63KiB | -0.1 | 99.78% | 16.32MiB | 226.42KiB | 4.62KiB | 0 | 0.0135477 | 16.3MiB | 138.19KiB | 2.83KiB | 0 | 0.00827711 | False | False |
| syslog_regex_logs2metric_ddmetrics | -20.45KiB | -0.16 | 77.76% | 12.48MiB | 631.02KiB | 12.85KiB | 0 | 0.0493534 | 12.46MiB | 527.67KiB | 10.76KiB | 0 | 0.0413363 | False | False |
| http_to_http_json | -44.1KiB | -0.18 | 99.91% | 23.85MiB | 332.23KiB | 6.78KiB | 0 | 0.0136014 | 23.81MiB | 559.23KiB | 11.42KiB | 0 | 0.0229363 | False | False |
| http_pipelines_no_grok_blackhole | -27.74KiB | -0.25 | 80.63% | 10.66MiB | 174.56KiB | 3.56KiB | 0 | 0.0159933 | 10.63MiB | 1.01MiB | 21.04KiB | 0 | 0.0950099 | False | False |
| fluent_elasticsearch | -246.33KiB | -0.3 | 99.99% | 79.47MiB | 53.4KiB | 1.08KiB | 0 | 0.000656008 | 79.23MiB | 3.1MiB | 63.84KiB | 0 | 0.0391454 | False | False |
| http_pipelines_blackhole | -5.54KiB | -0.32 | 97.31% | 1.67MiB | 49.27KiB | 1.01KiB | 0 | 0.0288052 | 1.66MiB | 112.36KiB | 2.29KiB | 0 | 0.065907 | False | False |
| http_to_http_noack | -86.97KiB | -0.36 | 99.99% | 23.85MiB | 251.7KiB | 5.15KiB | 0 | 0.0103058 | 23.76MiB | 1.02MiB | 21.23KiB | 0 | 0.0428237 | False | False |
| datadog_agent_remap_datadog_logs | -584.67KiB | -0.94 | 100.00% | 60.62MiB | 2.22MiB | 46.46KiB | 0 | 0.0365652 | 60.05MiB | 4.54MiB | 94.57KiB | 0 | 0.0756069 | False | False |
| datadog_agent_remap_datadog_logs_acks | -802.96KiB | -1.3 | 100.00% | 60.42MiB | 3.27MiB | 68.34KiB | 0 | 0.054103 | 59.63MiB | 4.21MiB | 87.72KiB | 0 | 0.070651 | False | False |
Soak Test Results
Baseline: dec33dc4956fb375fe889d5e41c1db911ec13835 Comparison: 39ed4d155e116bde1f8f4d512f02709ee4661a1a Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_loki | 531.83KiB | 3.63 | 100.00% | 14.29MiB | 363.07KiB | 7.43KiB | 0 | 0.0248007 | 14.81MiB | 748.63KiB | 15.22KiB | 0 | 0.0493445 | False | False |
| splunk_hec_route_s3 | 510.0KiB | 2.77 | 100.00% | 17.96MiB | 2.37MiB | 49.46KiB | 0 | 0.132162 | 18.46MiB | 2.29MiB | 47.9KiB | 0 | 0.124003 | False | False |
| datadog_agent_remap_datadog_logs | 1.46MiB | 2.4 | 100.00% | 60.72MiB | 1.39MiB | 29.16KiB | 0 | 0.0229029 | 62.18MiB | 4.47MiB | 93.01KiB | 0 | 0.071837 | False | False |
| datadog_agent_remap_blackhole_acks | 773.04KiB | 1.26 | 100.00% | 59.84MiB | 4.11MiB | 85.65KiB | 0 | 0.0687066 | 60.59MiB | 2.25MiB | 47.04KiB | 0 | 0.0370519 | False | False |
| http_to_http_acks | 175.36KiB | 1.01 | 55.14% | 17.02MiB | 7.79MiB | 162.91KiB | 0 | 0.457699 | 17.19MiB | 7.87MiB | 164.37KiB | 0 | 0.457774 | True | True |
| datadog_agent_remap_blackhole | 320.55KiB | 0.52 | 99.75% | 60.61MiB | 4.0MiB | 83.36KiB | 0 | 0.0659992 | 60.93MiB | 3.15MiB | 65.61KiB | 0 | 0.0516161 | False | False |
| syslog_humio_logs | 61.66KiB | 0.37 | 100.00% | 16.22MiB | 306.32KiB | 6.25KiB | 0 | 0.0184447 | 16.28MiB | 297.8KiB | 6.1KiB | 0 | 0.0178652 | False | False |
| http_pipelines_blackhole_acks | 3.97KiB | 0.32 | 86.38% | 1.2MiB | 101.73KiB | 2.07KiB | 0 | 0.0827037 | 1.2MiB | 82.22KiB | 1.68KiB | 0 | 0.0666272 | False | False |
| syslog_splunk_hec_logs | 37.33KiB | 0.22 | 92.11% | 16.37MiB | 765.12KiB | 15.57KiB | 0 | 0.0456294 | 16.41MiB | 709.26KiB | 14.45KiB | 0 | 0.0422039 | False | False |
| syslog_log2metric_splunk_hec_metrics | 35.56KiB | 0.2 | 90.54% | 17.38MiB | 698.19KiB | 14.22KiB | 0 | 0.0392181 | 17.42MiB | 777.11KiB | 15.82KiB | 0 | 0.0435639 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 21.31KiB | 0.09 | 91.26% | 23.82MiB | 510.31KiB | 10.41KiB | 0 | 0.0209208 | 23.84MiB | 335.49KiB | 6.85KiB | 0 | 0.0137417 | False | False |
| splunk_hec_indexer_ack_blackhole | 10.9KiB | 0.04 | 32.84% | 23.74MiB | 920.09KiB | 18.71KiB | 0 | 0.0378354 | 23.75MiB | 866.73KiB | 17.63KiB | 0 | 0.0356252 | False | False |
| enterprise_http_to_http | -1.35KiB | -0.01 | 14.83% | 23.85MiB | 251.59KiB | 5.14KiB | 0 | 0.0103009 | 23.85MiB | 249.5KiB | 5.11KiB | 0 | 0.0102159 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -6.4KiB | -0.03 | 20.27% | 23.75MiB | 854.38KiB | 17.38KiB | 0 | 0.0351204 | 23.75MiB | 877.78KiB | 17.85KiB | 0 | 0.0360915 | False | False |
| file_to_blackhole | -63.81KiB | -0.07 | 46.47% | 95.34MiB | 3.25MiB | 67.44KiB | 0 | 0.034116 | 95.28MiB | 3.74MiB | 77.76KiB | 0 | 0.0392408 | False | False |
| http_to_http_json | -34.97KiB | -0.14 | 99.44% | 23.85MiB | 332.51KiB | 6.79KiB | 0 | 0.0136135 | 23.81MiB | 520.05KiB | 10.62KiB | 0 | 0.0213222 | False | False |
| fluent_elasticsearch | -195.27KiB | -0.24 | 100.00% | 79.47MiB | 54.11KiB | 1.09KiB | 0 | 0.000664806 | 79.28MiB | 1.72MiB | 35.38KiB | 0 | 0.0216777 | False | False |
| http_to_http_noack | -78.52KiB | -0.32 | 99.86% | 23.83MiB | 517.61KiB | 10.57KiB | 0 | 0.0212084 | 23.75MiB | 1.06MiB | 22.19KiB | 0 | 0.0447947 | False | False |
| datadog_agent_remap_datadog_logs_acks | -288.65KiB | -0.45 | 99.02% | 62.11MiB | 3.12MiB | 65.26KiB | 0 | 0.050237 | 61.83MiB | 4.36MiB | 90.71KiB | 0 | 0.0704527 | False | False |
| syslog_log2metric_humio_metrics | -77.42KiB | -0.61 | 100.00% | 12.39MiB | 276.17KiB | 5.64KiB | 0 | 0.0217706 | 12.31MiB | 553.88KiB | 11.27KiB | 0 | 0.043931 | False | False |
| http_pipelines_no_grok_blackhole | -77.24KiB | -0.7 | 99.95% | 10.76MiB | 81.96KiB | 1.67KiB | 0 | 0.0074347 | 10.69MiB | 1.06MiB | 22.12KiB | 0 | 0.0993603 | False | False |
| http_pipelines_blackhole | -14.79KiB | -0.88 | 100.00% | 1.65MiB | 38.39KiB | 803.82B | 0 | 0.0227393 | 1.63MiB | 122.36KiB | 2.49KiB | 0 | 0.0731133 | False | False |
| socket_to_socket_blackhole | -215.67KiB | -0.91 | 100.00% | 23.18MiB | 824.8KiB | 16.84KiB | 0 | 0.0347452 | 22.97MiB | 849.25KiB | 17.34KiB | 0 | 0.0361033 | False | False |
| http_text_to_http_json | -419.47KiB | -1.04 | 100.00% | 39.46MiB | 901.19KiB | 18.4KiB | 0 | 0.0222972 | 39.05MiB | 944.59KiB | 19.29KiB | 0 | 0.0236162 | False | False |
| syslog_regex_logs2metric_ddmetrics | -190.68KiB | -1.5 | 100.00% | 12.39MiB | 562.76KiB | 11.47KiB | 0 | 0.0443406 | 12.21MiB | 582.59KiB | 11.88KiB | 0 | 0.0466034 | False | False |
Soak Test Results
Baseline: 18aead8d070976a64313939191fb9635623fc69e Comparison: 0246402f40c850705cd1cb2b19b79849ef68bada Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_splunk_hec_logs | 345.9KiB | 2.11 | 100.00% | 16.04MiB | 977.27KiB | 19.89KiB | 0 | 0.0594949 | 16.38MiB | 705.84KiB | 14.39KiB | 0 | 0.0420844 | False | False |
| syslog_log2metric_humio_metrics | 167.72KiB | 1.32 | 100.00% | 12.4MiB | 551.61KiB | 11.26KiB | 0 | 0.0434244 | 12.57MiB | 573.57KiB | 11.67KiB | 0 | 0.0445646 | False | False |
| splunk_hec_indexer_ack_blackhole | 18.76KiB | 0.08 | 50.80% | 23.73MiB | 983.64KiB | 20.0KiB | 0 | 0.0404664 | 23.75MiB | 913.53KiB | 18.59KiB | 0 | 0.037553 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 11.74KiB | 0.05 | 70.52% | 23.83MiB | 434.83KiB | 8.87KiB | 0 | 0.017818 | 23.84MiB | 335.11KiB | 6.84KiB | 0 | 0.013725 | False | False |
| enterprise_http_to_http | 25.95B | 0 | 0.28% | 23.85MiB | 246.87KiB | 5.04KiB | 0 | 0.0101081 | 23.85MiB | 252.47KiB | 5.17KiB | 0 | 0.0103374 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -3.05KiB | -0.01 | 9.34% | 23.76MiB | 906.44KiB | 18.44KiB | 0 | 0.0372524 | 23.75MiB | 903.8KiB | 18.38KiB | 0 | 0.0371487 | False | False |
| http_to_http_json | -22.3KiB | -0.09 | 94.11% | 23.84MiB | 346.73KiB | 7.08KiB | 0 | 0.0141984 | 23.82MiB | 462.08KiB | 9.45KiB | 0 | 0.0189392 | False | False |
| fluent_elasticsearch | -91.83KiB | -0.11 | 100.00% | 79.47MiB | 52.42KiB | 1.06KiB | 0 | 0.00064395 | 79.38MiB | 845.42KiB | 16.99KiB | 0 | 0.0103981 | False | False |
| file_to_blackhole | -122.9KiB | -0.13 | 75.82% | 95.42MiB | 3.55MiB | 73.49KiB | 0 | 0.0371437 | 95.3MiB | 3.6MiB | 74.96KiB | 0 | 0.0377951 | False | False |
| socket_to_socket_blackhole | -41.41KiB | -0.18 | 100.00% | 22.58MiB | 313.42KiB | 6.4KiB | 0 | 0.0135539 | 22.54MiB | 371.97KiB | 7.59KiB | 0 | 0.016115 | False | False |
| syslog_regex_logs2metric_ddmetrics | -25.29KiB | -0.2 | 90.77% | 12.38MiB | 528.24KiB | 10.76KiB | 0 | 0.0416506 | 12.36MiB | 514.09KiB | 10.48KiB | 0 | 0.040616 | False | False |
| splunk_hec_route_s3 | -42.13KiB | -0.25 | 34.42% | 16.15MiB | 3.19MiB | 66.47KiB | 0 | 0.197584 | 16.1MiB | 3.21MiB | 67.19KiB | 0 | 0.199437 | False | False |
| http_pipelines_blackhole | -5.39KiB | -0.31 | 93.67% | 1.68MiB | 57.7KiB | 1.18KiB | 0 | 0.0336105 | 1.67MiB | 130.25KiB | 2.65KiB | 0 | 0.0761057 | False | False |
| http_to_http_noack | -122.26KiB | -0.5 | 100.00% | 23.84MiB | 407.28KiB | 8.32KiB | 0 | 0.0166821 | 23.72MiB | 1.23MiB | 25.67KiB | 0 | 0.0519232 | False | False |
| http_text_to_http_json | -202.28KiB | -0.52 | 100.00% | 38.04MiB | 854.36KiB | 17.44KiB | 0 | 0.0219266 | 37.85MiB | 898.02KiB | 18.33KiB | 0 | 0.0231674 | False | False |
| datadog_agent_remap_datadog_logs | -491.23KiB | -0.78 | 100.00% | 61.31MiB | 288.61KiB | 5.91KiB | 0 | 0.00459569 | 60.83MiB | 4.21MiB | 87.58KiB | 0 | 0.0691305 | False | False |
| datadog_agent_remap_datadog_logs_acks | -494.72KiB | -0.8 | 100.00% | 60.71MiB | 2.19MiB | 45.82KiB | 0 | 0.0360227 | 60.23MiB | 4.27MiB | 88.92KiB | 0 | 0.0709073 | False | False |
| datadog_agent_remap_blackhole | -568.23KiB | -0.91 | 100.00% | 61.11MiB | 4.27MiB | 88.96KiB | 0 | 0.0698888 | 60.56MiB | 3.31MiB | 68.95KiB | 0 | 0.0545856 | False | False |
| http_to_http_acks | -166.14KiB | -0.93 | 52.97% | 17.51MiB | 7.43MiB | 155.35KiB | 0 | 0.424353 | 17.35MiB | 8.13MiB | 169.73KiB | 0 | 0.46856 | True | True |
| http_pipelines_blackhole_acks | -14.26KiB | -1.14 | 100.00% | 1.22MiB | 94.53KiB | 1.93KiB | 0 | 0.0754701 | 1.21MiB | 78.75KiB | 1.61KiB | 0 | 0.0635958 | False | False |
| syslog_humio_logs | -268.34KiB | -1.6 | 100.00% | 16.37MiB | 583.1KiB | 11.9KiB | 0 | 0.0347826 | 16.11MiB | 730.25KiB | 14.96KiB | 0 | 0.0442686 | False | False |
| datadog_agent_remap_blackhole_acks | -1.02MiB | -1.72 | 100.00% | 59.62MiB | 4.99MiB | 103.97KiB | 0 | 0.0837105 | 58.59MiB | 4.59MiB | 96.08KiB | 0 | 0.0783877 | False | False |
| http_pipelines_no_grok_blackhole | -207.87KiB | -1.88 | 100.00% | 10.79MiB | 89.23KiB | 1.82KiB | 0 | 0.0080721 | 10.59MiB | 1.08MiB | 22.55KiB | 0 | 0.102273 | False | False |
| syslog_loki | -302.65KiB | -2.03 | 100.00% | 14.55MiB | 551.01KiB | 11.27KiB | 0 | 0.0369641 | 14.26MiB | 724.36KiB | 14.72KiB | 0 | 0.0496009 | False | False |
| syslog_log2metric_splunk_hec_metrics | -648.74KiB | -3.57 | 100.00% | 17.74MiB | 470.58KiB | 9.6KiB | 0 | 0.0258937 | 17.11MiB | 1.09MiB | 22.74KiB | 0 | 0.0637154 | False | False |
Soak Test Results
Baseline: f19053c3fb46b4c79c1ca2f43af11027c25229c7 Comparison: 701bc58d397fedd66a29fc53068a78624e8fe53c Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_datadog_logs_acks | 1.65MiB | 2.96 | 100.00% | 55.73MiB | 6.29MiB | 131.75KiB | 0 | 0.112886 | 57.38MiB | 6.54MiB | 136.05KiB | 0 | 0.113877 | False | False |
| socket_to_socket_blackhole | 486.63KiB | 2.2 | 100.00% | 21.58MiB | 468.22KiB | 9.56KiB | 0 | 0.0211824 | 22.06MiB | 410.49KiB | 8.38KiB | 0 | 0.0181707 | False | False |
| http_text_to_http_json | 636.08KiB | 1.69 | 100.00% | 36.83MiB | 874.11KiB | 17.84KiB | 0 | 0.0231753 | 37.45MiB | 952.39KiB | 19.44KiB | 0 | 0.0248319 | False | False |
| syslog_loki | 150.8KiB | 1.05 | 100.00% | 14.0MiB | 241.76KiB | 4.95KiB | 0 | 0.0168568 | 14.15MiB | 711.33KiB | 14.46KiB | 0 | 0.0490815 | False | False |
| splunk_hec_route_s3 | 127.35KiB | 0.7 | 93.76% | 17.76MiB | 2.36MiB | 49.05KiB | 0 | 0.132651 | 17.88MiB | 2.28MiB | 47.59KiB | 0 | 0.127253 | False | False |
| http_pipelines_blackhole_acks | 6.98KiB | 0.55 | 98.50% | 1.24MiB | 107.59KiB | 2.19KiB | 0 | 0.0850022 | 1.24MiB | 90.9KiB | 1.85KiB | 0 | 0.0714229 | False | False |
| datadog_agent_remap_datadog_logs | 123.89KiB | 0.2 | 86.89% | 60.03MiB | 363.49KiB | 7.44KiB | 0 | 0.00591176 | 60.15MiB | 3.92MiB | 81.7KiB | 0 | 0.0651929 | False | False |
| http_pipelines_blackhole | 1.95KiB | 0.11 | 63.85% | 1.68MiB | 26.84KiB | 561.9B | 0 | 0.0155928 | 1.68MiB | 101.52KiB | 2.07KiB | 0 | 0.0589191 | False | False |
| splunk_hec_indexer_ack_blackhole | 16.63KiB | 0.07 | 47.74% | 23.74MiB | 928.98KiB | 18.89KiB | 0 | 0.0382024 | 23.76MiB | 879.0KiB | 17.89KiB | 0 | 0.0361225 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 12.31KiB | 0.05 | 72.86% | 23.83MiB | 430.88KiB | 8.8KiB | 0 | 0.0176564 | 23.84MiB | 338.24KiB | 6.9KiB | 0 | 0.0138532 | False | False |
| enterprise_http_to_http | -2.11KiB | -0.01 | 21.62% | 23.85MiB | 265.41KiB | 5.42KiB | 0 | 0.0108665 | 23.85MiB | 266.77KiB | 5.46KiB | 0 | 0.0109229 | False | False |
| file_to_blackhole | -44.94KiB | -0.05 | 30.42% | 95.31MiB | 3.82MiB | 79.05KiB | 0 | 0.0400399 | 95.27MiB | 4.01MiB | 83.45KiB | 0 | 0.0421332 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -15.85KiB | -0.07 | 47.55% | 23.76MiB | 839.47KiB | 17.08KiB | 0 | 0.0344992 | 23.74MiB | 891.45KiB | 18.13KiB | 0 | 0.0366594 | False | False |
| http_to_http_json | -37.27KiB | -0.15 | 99.55% | 23.84MiB | 349.29KiB | 7.13KiB | 0 | 0.0143031 | 23.81MiB | 539.23KiB | 11.0KiB | 0 | 0.0221145 | False | False |
| fluent_elasticsearch | -247.91KiB | -0.3 | 100.00% | 79.47MiB | 53.33KiB | 1.08KiB | 0 | 0.00065518 | 79.23MiB | 2.01MiB | 41.37KiB | 0 | 0.0253925 | False | False |
| http_to_http_noack | -104.4KiB | -0.43 | 100.00% | 23.84MiB | 255.78KiB | 5.23KiB | 0 | 0.0104731 | 23.74MiB | 1.09MiB | 22.83KiB | 0 | 0.0461 | False | False |
| http_pipelines_no_grok_blackhole | -54.84KiB | -0.5 | 98.55% | 10.71MiB | 140.4KiB | 2.87KiB | 0 | 0.0128006 | 10.66MiB | 1.07MiB | 22.24KiB | 0 | 0.100226 | False | False |
| syslog_regex_logs2metric_ddmetrics | -111.09KiB | -0.87 | 100.00% | 12.44MiB | 587.22KiB | 11.96KiB | 0 | 0.0460855 | 12.33MiB | 398.72KiB | 8.14KiB | 0 | 0.0315675 | False | False |
| datadog_agent_remap_blackhole | -595.15KiB | -1 | 100.00% | 58.35MiB | 4.29MiB | 89.32KiB | 0 | 0.0734557 | 57.77MiB | 3.53MiB | 73.62KiB | 0 | 0.0610454 | False | False |
| syslog_log2metric_humio_metrics | -168.55KiB | -1.32 | 100.00% | 12.49MiB | 371.84KiB | 7.59KiB | 0 | 0.0290691 | 12.32MiB | 538.51KiB | 10.96KiB | 0 | 0.0426604 | False | False |
| http_to_http_acks | -258.7KiB | -1.45 | 71.85% | 17.39MiB | 8.16MiB | 170.74KiB | 0 | 0.469445 | 17.14MiB | 8.09MiB | 168.97KiB | 0 | 0.472021 | True | True |
| syslog_splunk_hec_logs | -244.02KiB | -1.48 | 100.00% | 16.09MiB | 804.1KiB | 16.35KiB | 0 | 0.048803 | 15.85MiB | 628.88KiB | 12.82KiB | 0 | 0.0387424 | False | False |
| datadog_agent_remap_blackhole_acks | -1011.42KiB | -1.63 | 100.00% | 60.71MiB | 4.26MiB | 88.76KiB | 0 | 0.0702028 | 59.73MiB | 2.84MiB | 59.3KiB | 0 | 0.047488 | False | False |
| syslog_humio_logs | -452.94KiB | -2.76 | 100.00% | 16.0MiB | 285.34KiB | 5.82KiB | 0 | 0.0174098 | 15.56MiB | 217.77KiB | 4.46KiB | 0 | 0.0136647 | False | False |
| syslog_log2metric_splunk_hec_metrics | -604.66KiB | -3.33 | 100.00% | 17.73MiB | 527.99KiB | 10.76KiB | 0 | 0.0290791 | 17.14MiB | 825.84KiB | 16.81KiB | 0 | 0.0470506 | False | False |
Soak Test Results
Baseline: 52175fdf0f05b736c6e194df127b6334bdd7c5b9 Comparison: 62871caf6b62545598aa7710387d06e0e810b2ad Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| socket_to_socket_blackhole | 764.59KiB | 3.35 | 100.00% | 22.31MiB | 211.54KiB | 4.32KiB | 0 | 0.0092589 | 23.05MiB | 199.95KiB | 4.08KiB | 0 | 0.00846818 | False | False |
| syslog_loki | 458.12KiB | 3.13 | 100.00% | 14.29MiB | 344.37KiB | 7.05KiB | 0 | 0.0235315 | 14.74MiB | 761.12KiB | 15.47KiB | 0 | 0.0504308 | False | False |
| http_text_to_http_json | 596.78KiB | 1.57 | 100.00% | 37.0MiB | 877.11KiB | 17.9KiB | 0 | 0.0231435 | 37.59MiB | 944.67KiB | 19.28KiB | 0 | 0.0245397 | False | False |
| http_pipelines_blackhole_acks | 11.45KiB | 0.99 | 99.99% | 1.13MiB | 124.13KiB | 2.53KiB | 0 | 0.106934 | 1.14MiB | 76.23KiB | 1.56KiB | 0 | 0.0650284 | False | False |
| datadog_agent_remap_datadog_logs_acks | 414.65KiB | 0.66 | 99.98% | 61.5MiB | 3.11MiB | 65.03KiB | 0 | 0.0505729 | 61.91MiB | 4.4MiB | 91.61KiB | 0 | 0.0710798 | False | False |
| datadog_agent_remap_blackhole | 332.97KiB | 0.56 | 98.85% | 57.86MiB | 5.06MiB | 105.39KiB | 0 | 0.0874128 | 58.18MiB | 3.78MiB | 78.88KiB | 0 | 0.0649103 | False | False |
| syslog_splunk_hec_logs | 57.18KiB | 0.35 | 99.83% | 15.87MiB | 737.85KiB | 15.01KiB | 0 | 0.0453819 | 15.93MiB | 507.03KiB | 10.36KiB | 0 | 0.0310761 | False | False |
| datadog_agent_remap_datadog_logs | 210.43KiB | 0.34 | 98.45% | 60.63MiB | 1.18MiB | 24.62KiB | 0 | 0.0193789 | 60.83MiB | 4.0MiB | 83.37KiB | 0 | 0.0657798 | False | False |
| splunk_hec_route_s3 | 43.84KiB | 0.24 | 48.30% | 17.73MiB | 2.31MiB | 48.15KiB | 0 | 0.130456 | 17.77MiB | 2.27MiB | 47.52KiB | 0 | 0.127812 | False | False |
| syslog_regex_logs2metric_ddmetrics | 28.55KiB | 0.23 | 94.79% | 12.29MiB | 511.53KiB | 10.42KiB | 0 | 0.0406311 | 12.32MiB | 508.29KiB | 10.36KiB | 0 | 0.0402827 | False | False |
| syslog_humio_logs | 34.31KiB | 0.21 | 100.00% | 16.24MiB | 195.75KiB | 4.0KiB | 0 | 0.011765 | 16.28MiB | 181.9KiB | 3.72KiB | 0 | 0.0109103 | False | False |
| http_to_http_acks | 27.51KiB | 0.16 | 9.23% | 17.09MiB | 8.05MiB | 168.21KiB | 0 | 0.47086 | 17.11MiB | 8.02MiB | 167.38KiB | 0 | 0.468294 | True | True |
| datadog_agent_remap_blackhole_acks | 86.89KiB | 0.14 | 59.57% | 60.01MiB | 4.1MiB | 85.42KiB | 0 | 0.0683118 | 60.1MiB | 2.85MiB | 59.62KiB | 0 | 0.0474003 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 11.07KiB | 0.05 | 67.93% | 23.83MiB | 433.13KiB | 8.84KiB | 0 | 0.0177499 | 23.84MiB | 332.58KiB | 6.79KiB | 0 | 0.0136231 | False | False |
| syslog_log2metric_splunk_hec_metrics | 4.54KiB | 0.03 | 18.89% | 17.51MiB | 632.86KiB | 12.89KiB | 0 | 0.0352816 | 17.52MiB | 684.03KiB | 13.93KiB | 0 | 0.0381251 | False | False |
| splunk_hec_indexer_ack_blackhole | -226.74B | -0 | 0.71% | 23.75MiB | 861.88KiB | 17.53KiB | 0 | 0.0354285 | 23.75MiB | 861.34KiB | 17.52KiB | 0 | 0.0354069 | False | False |
| enterprise_http_to_http | -1.64KiB | -0.01 | 18.11% | 23.85MiB | 245.29KiB | 5.01KiB | 0 | 0.0100431 | 23.84MiB | 248.56KiB | 5.09KiB | 0 | 0.0101776 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -2.61KiB | -0.01 | 8.17% | 23.75MiB | 887.9KiB | 18.06KiB | 0 | 0.0364975 | 23.75MiB | 879.59KiB | 17.89KiB | 0 | 0.0361597 | False | False |
| syslog_log2metric_humio_metrics | -3.86KiB | -0.03 | 23.90% | 12.64MiB | 344.39KiB | 7.02KiB | 0 | 0.0266064 | 12.63MiB | 519.1KiB | 10.57KiB | 0 | 0.040116 | False | False |
| file_to_blackhole | -43.73KiB | -0.04 | 32.96% | 95.34MiB | 3.16MiB | 65.51KiB | 0 | 0.0331374 | 95.3MiB | 3.81MiB | 79.16KiB | 0 | 0.0399195 | False | False |
| http_to_http_json | -31.0KiB | -0.13 | 98.84% | 23.85MiB | 330.14KiB | 6.74KiB | 0 | 0.0135168 | 23.82MiB | 502.34KiB | 10.27KiB | 0 | 0.0205933 | False | False |
| fluent_elasticsearch | -191.92KiB | -0.24 | 100.00% | 79.47MiB | 52.57KiB | 1.06KiB | 0 | 0.000645897 | 79.29MiB | 1.61MiB | 33.12KiB | 0 | 0.0203046 | False | False |
| http_pipelines_blackhole | -4.1KiB | -0.24 | 88.12% | 1.69MiB | 12.8KiB | 268.15B | 0 | 0.00739278 | 1.69MiB | 128.43KiB | 2.62KiB | 0 | 0.0743542 | False | False |
| http_to_http_noack | -131.21KiB | -0.54 | 100.00% | 23.85MiB | 249.03KiB | 5.09KiB | 0 | 0.0101964 | 23.72MiB | 1.23MiB | 25.66KiB | 0 | 0.0519087 | False | False |
| http_pipelines_no_grok_blackhole | -119.74KiB | -1.08 | 100.00% | 10.86MiB | 75.37KiB | 1.54KiB | 0 | 0.00677781 | 10.74MiB | 1.07MiB | 22.24KiB | 0 | 0.0994258 | False | False |
Soak Test Results
Baseline: 11905070fbe0795bdc975995269f17208e8041b2 Comparison: 3c91705e563c726963c377108a9a9ea40e293710 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| socket_to_socket_blackhole | 508.87KiB | 2.2 | 100.00% | 22.55MiB | 838.25KiB | 17.11KiB | 0 | 0.0363013 | 23.04MiB | 945.19KiB | 19.29KiB | 0 | 0.0400496 | False | False |
| http_text_to_http_json | 701.78KiB | 1.78 | 100.00% | 38.56MiB | 858.2KiB | 17.51KiB | 0 | 0.0217327 | 39.24MiB | 863.57KiB | 17.63KiB | 0 | 0.0214868 | False | False |
| http_pipelines_blackhole_acks | 15.57KiB | 1.26 | 100.00% | 1.21MiB | 108.53KiB | 2.21KiB | 0 | 0.0877296 | 1.22MiB | 57.36KiB | 1.17KiB | 0 | 0.04579 | False | False |
| splunk_hec_route_s3 | 108.31KiB | 0.57 | 89.80% | 18.47MiB | 2.27MiB | 47.36KiB | 0 | 0.123078 | 18.58MiB | 2.21MiB | 46.3KiB | 0 | 0.119119 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 21.77KiB | 0.09 | 92.36% | 23.82MiB | 500.1KiB | 10.21KiB | 0 | 0.0205008 | 23.84MiB | 334.71KiB | 6.83KiB | 0 | 0.0137089 | False | False |
| splunk_hec_indexer_ack_blackhole | 11.21KiB | 0.05 | 33.40% | 23.74MiB | 916.02KiB | 18.63KiB | 0 | 0.0376685 | 23.75MiB | 889.71KiB | 18.1KiB | 0 | 0.0365695 | False | False |
| enterprise_http_to_http | 615.42B | 0 | 6.56% | 23.84MiB | 252.75KiB | 5.16KiB | 0 | 0.0103493 | 23.85MiB | 252.29KiB | 5.16KiB | 0 | 0.01033 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -7.84KiB | -0.03 | 24.54% | 23.75MiB | 859.52KiB | 17.48KiB | 0 | 0.0353321 | 23.74MiB | 883.91KiB | 17.98KiB | 0 | 0.0363464 | False | False |
| file_to_blackhole | -64.04KiB | -0.07 | 46.01% | 95.34MiB | 3.3MiB | 68.34KiB | 0 | 0.0345697 | 95.28MiB | 3.8MiB | 79.03KiB | 0 | 0.0398985 | False | False |
| http_to_http_json | -34.11KiB | -0.14 | 99.25% | 23.84MiB | 340.17KiB | 6.94KiB | 0 | 0.0139293 | 23.81MiB | 523.71KiB | 10.69KiB | 0 | 0.021475 | False | False |
| syslog_loki | -48.53KiB | -0.33 | 99.72% | 14.41MiB | 321.65KiB | 6.58KiB | 0 | 0.0217947 | 14.36MiB | 729.91KiB | 14.84KiB | 0 | 0.0496217 | False | False |
| http_pipelines_blackhole | -6.39KiB | -0.37 | 99.86% | 1.69MiB | 12.91KiB | 269.94B | 0 | 0.00745623 | 1.68MiB | 96.91KiB | 1.98KiB | 0 | 0.0561947 | False | False |
| http_to_http_noack | -104.93KiB | -0.43 | 100.00% | 23.84MiB | 407.25KiB | 8.33KiB | 0 | 0.0166808 | 23.73MiB | 1.15MiB | 24.01KiB | 0 | 0.0485118 | False | False |
| fluent_elasticsearch | -446.85KiB | -0.55 | 100.00% | 79.47MiB | 52.95KiB | 1.07KiB | 0 | 0.000650495 | 79.04MiB | 4.56MiB | 93.72KiB | 0 | 0.0577046 | False | False |
| datadog_agent_remap_blackhole | -629.87KiB | -1.03 | 100.00% | 60.0MiB | 3.75MiB | 78.19KiB | 0 | 0.0625202 | 59.38MiB | 3.1MiB | 64.6KiB | 0 | 0.0521287 | False | False |
| datadog_agent_remap_datadog_logs_acks | -644.35KiB | -1.04 | 100.00% | 60.47MiB | 2.57MiB | 53.85KiB | 0 | 0.0425393 | 59.84MiB | 4.23MiB | 87.98KiB | 0 | 0.0706159 | False | False |
| datadog_agent_remap_datadog_logs | -753.85KiB | -1.23 | 100.00% | 59.68MiB | 1.13MiB | 23.65KiB | 0 | 0.0188918 | 58.95MiB | 4.04MiB | 84.17KiB | 0 | 0.0685538 | False | False |
| datadog_agent_remap_blackhole_acks | -911.07KiB | -1.47 | 100.00% | 60.63MiB | 4.54MiB | 94.56KiB | 0 | 0.0749092 | 59.74MiB | 3.12MiB | 65.13KiB | 0 | 0.052146 | False | False |
| syslog_regex_logs2metric_ddmetrics | -221.93KiB | -1.75 | 100.00% | 12.4MiB | 481.02KiB | 9.81KiB | 0 | 0.0378851 | 12.18MiB | 559.52KiB | 11.39KiB | 0 | 0.0448521 | False | False |
| syslog_splunk_hec_logs | -293.62KiB | -1.76 | 100.00% | 16.28MiB | 889.53KiB | 18.09KiB | 0 | 0.053346 | 15.99MiB | 844.47KiB | 17.21KiB | 0 | 0.0515515 | False | False |
| syslog_humio_logs | -314.99KiB | -1.92 | 100.00% | 16.03MiB | 434.93KiB | 8.88KiB | 0 | 0.0264894 | 15.72MiB | 464.32KiB | 9.51KiB | 0 | 0.0288328 | False | False |
| http_pipelines_no_grok_blackhole | -231.47KiB | -2.08 | 100.00% | 10.87MiB | 41.93KiB | 876.42B | 0 | 0.00376777 | 10.64MiB | 1.12MiB | 23.31KiB | 0 | 0.105213 | False | False |
| http_to_http_acks | -449.42KiB | -2.52 | 94.13% | 17.43MiB | 8.06MiB | 168.44KiB | 0 | 0.462203 | 16.99MiB | 8.03MiB | 167.71KiB | 0 | 0.472675 | True | True |
| syslog_log2metric_splunk_hec_metrics | -535.48KiB | -2.98 | 100.00% | 17.53MiB | 579.39KiB | 11.82KiB | 0 | 0.0322689 | 17.01MiB | 941.65KiB | 19.17KiB | 0 | 0.0540572 | False | False |
| syslog_log2metric_humio_metrics | -466.04KiB | -3.55 | 100.00% | 12.83MiB | 313.69KiB | 6.4KiB | 0 | 0.0238754 | 12.37MiB | 528.02KiB | 10.75KiB | 0 | 0.0416661 | False | False |
Soak Test Results
Baseline: e892bbdcc40e6595896c1e2e30307a3469b25be5 Comparison: 3769c44ba4698ebf8b9d525fe7bd4b328877fd3d Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| socket_to_socket_blackhole | 459.04KiB | 1.95 | 100.00% | 23.04MiB | 181.74KiB | 3.71KiB | 0 | 0.00770221 | 23.49MiB | 286.98KiB | 5.86KiB | 0 | 0.0119301 | False | False |
| http_text_to_http_json | 662.17KiB | 1.68 | 100.00% | 38.47MiB | 863.05KiB | 17.62KiB | 0 | 0.0219056 | 39.11MiB | 824.24KiB | 16.83KiB | 0 | 0.0205746 | False | False |
| http_pipelines_blackhole_acks | 13.19KiB | 1.09 | 100.00% | 1.18MiB | 118.67KiB | 2.41KiB | 0 | 0.0982579 | 1.19MiB | 99.8KiB | 2.03KiB | 0 | 0.0817426 | False | False |
| splunk_hec_route_s3 | 164.01KiB | 0.86 | 98.75% | 18.72MiB | 2.25MiB | 46.95KiB | 0 | 0.120378 | 18.88MiB | 2.19MiB | 45.87KiB | 0 | 0.116115 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 30.2KiB | 0.12 | 97.76% | 23.81MiB | 550.46KiB | 11.23KiB | 0 | 0.0225743 | 23.84MiB | 341.77KiB | 6.98KiB | 0 | 0.0139988 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 8.08KiB | 0.03 | 25.75% | 23.75MiB | 868.86KiB | 17.67KiB | 0 | 0.0357216 | 23.76MiB | 841.13KiB | 17.11KiB | 0 | 0.0345701 | False | False |
| enterprise_http_to_http | 957.86B | 0 | 10.17% | 23.85MiB | 255.1KiB | 5.21KiB | 0 | 0.0104452 | 23.85MiB | 251.22KiB | 5.14KiB | 0 | 0.0102861 | False | False |
| splunk_hec_indexer_ack_blackhole | -516.68B | -0 | 1.49% | 23.75MiB | 937.65KiB | 19.07KiB | 0 | 0.038554 | 23.75MiB | 937.95KiB | 19.08KiB | 0 | 0.0385669 | False | False |
| file_to_blackhole | -57.28KiB | -0.06 | 45.47% | 95.34MiB | 3.06MiB | 63.48KiB | 0 | 0.03211 | 95.28MiB | 3.38MiB | 70.28KiB | 0 | 0.0354716 | False | False |
| http_to_http_json | -26.31KiB | -0.11 | 96.95% | 23.84MiB | 344.82KiB | 7.04KiB | 0 | 0.0141199 | 23.82MiB | 485.19KiB | 9.92KiB | 0 | 0.0198896 | False | False |
| fluent_elasticsearch | -175.51KiB | -0.22 | 100.00% | 79.47MiB | 52.21KiB | 1.06KiB | 0 | 0.000641388 | 79.3MiB | 1.63MiB | 33.5KiB | 0 | 0.0205139 | False | False |
| http_to_http_noack | -56.95KiB | -0.23 | 99.79% | 23.84MiB | 260.51KiB | 5.33KiB | 0 | 0.0106676 | 23.79MiB | 871.09KiB | 17.76KiB | 0 | 0.0357531 | False | False |
| http_pipelines_no_grok_blackhole | -56.63KiB | -0.52 | 99.14% | 10.68MiB | 96.8KiB | 1.98KiB | 0 | 0.00885282 | 10.62MiB | 1.03MiB | 21.44KiB | 0 | 0.0969054 | False | False |
| syslog_loki | -98.58KiB | -0.66 | 100.00% | 14.7MiB | 130.8KiB | 2.68KiB | 0 | 0.00869021 | 14.6MiB | 734.15KiB | 14.92KiB | 0 | 0.049096 | False | False |
| datadog_agent_remap_datadog_logs_acks | -543.32KiB | -0.86 | 100.00% | 61.57MiB | 3.3MiB | 68.95KiB | 0 | 0.0535864 | 61.04MiB | 4.38MiB | 91.17KiB | 0 | 0.0717414 | False | False |
| syslog_splunk_hec_logs | -157.7KiB | -0.95 | 100.00% | 16.18MiB | 721.11KiB | 14.68KiB | 0 | 0.0435217 | 16.02MiB | 571.72KiB | 11.66KiB | 0 | 0.034837 | False | False |
| http_to_http_acks | -224.45KiB | -1.27 | 66.04% | 17.29MiB | 8.04MiB | 168.05KiB | 0 | 0.46477 | 17.07MiB | 7.87MiB | 164.31KiB | 0 | 0.460843 | True | True |
| syslog_regex_logs2metric_ddmetrics | -184.77KiB | -1.45 | 100.00% | 12.47MiB | 554.96KiB | 11.31KiB | 0 | 0.0434608 | 12.29MiB | 511.86KiB | 10.43KiB | 0 | 0.0406739 | False | False |
| datadog_agent_remap_blackhole | -1003.21KiB | -1.62 | 100.00% | 60.57MiB | 3.94MiB | 82.16KiB | 0 | 0.0650699 | 59.59MiB | 3.19MiB | 66.52KiB | 0 | 0.0535216 | False | False |
| syslog_humio_logs | -288.23KiB | -1.71 | 100.00% | 16.46MiB | 213.47KiB | 4.36KiB | 0 | 0.0126601 | 16.18MiB | 104.99KiB | 2.15KiB | 0 | 0.0063346 | False | False |
| datadog_agent_remap_datadog_logs | -1.12MiB | -1.83 | 100.00% | 61.25MiB | 654.37KiB | 13.41KiB | 0 | 0.0104311 | 60.13MiB | 3.91MiB | 81.5KiB | 0 | 0.065062 | False | False |
| http_pipelines_blackhole | -31.55KiB | -1.88 | 100.00% | 1.64MiB | 72.27KiB | 1.48KiB | 0 | 0.0429852 | 1.61MiB | 134.62KiB | 2.74KiB | 0 | 0.0815959 | False | False |
| syslog_log2metric_splunk_hec_metrics | -398.62KiB | -2.25 | 100.00% | 17.28MiB | 662.64KiB | 13.51KiB | 0 | 0.037449 | 16.89MiB | 816.3KiB | 16.61KiB | 0 | 0.0471966 | False | False |
| datadog_agent_remap_blackhole_acks | -1.55MiB | -2.52 | 100.00% | 61.56MiB | 4.31MiB | 89.74KiB | 0 | 0.0700048 | 60.01MiB | 3.96MiB | 82.93KiB | 0 | 0.0660473 | False | False |
| syslog_log2metric_humio_metrics | -389.7KiB | -2.97 | 100.00% | 12.82MiB | 307.47KiB | 6.27KiB | 0 | 0.0234236 | 12.44MiB | 490.26KiB | 9.98KiB | 0 | 0.0384922 | False | False |
Soak Test Results
Baseline: cf7843b1284afc6542287af5c379cca52db07761 Comparison: 0c6b75ae4c59991926ee90853aae1f6b1fb3d99b Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_text_to_http_json | 1.23MiB | 3.21 | 100.00% | 38.37MiB | 838.05KiB | 17.11KiB | 0 | 0.0213246 | 39.6MiB | 751.74KiB | 15.36KiB | 0 | 0.0185339 | False | False |
| splunk_hec_route_s3 | 254.28KiB | 1.45 | 99.83% | 17.07MiB | 2.73MiB | 56.93KiB | 0 | 0.160203 | 17.32MiB | 2.76MiB | 57.74KiB | 0 | 0.159425 | False | False |
| syslog_loki | 207.38KiB | 1.44 | 100.00% | 14.07MiB | 295.12KiB | 6.04KiB | 0 | 0.0204819 | 14.27MiB | 706.99KiB | 14.37KiB | 0 | 0.0483706 | False | False |
| syslog_humio_logs | 219.76KiB | 1.43 | 100.00% | 14.97MiB | 1.43MiB | 29.8KiB | 0 | 0.0951525 | 15.19MiB | 1.4MiB | 29.32KiB | 0 | 0.092043 | False | False |
| syslog_regex_logs2metric_ddmetrics | 144.12KiB | 1.14 | 100.00% | 12.29MiB | 500.47KiB | 10.2KiB | 0 | 0.0397535 | 12.43MiB | 583.2KiB | 11.88KiB | 0 | 0.0457999 | False | False |
| http_pipelines_blackhole_acks | 11.9KiB | 0.98 | 99.99% | 1.18MiB | 114.25KiB | 2.32KiB | 0 | 0.0942151 | 1.2MiB | 87.63KiB | 1.79KiB | 0 | 0.0715603 | False | False |
| syslog_log2metric_splunk_hec_metrics | 132.5KiB | 0.77 | 100.00% | 16.81MiB | 724.78KiB | 14.78KiB | 0 | 0.0420886 | 16.94MiB | 818.25KiB | 16.68KiB | 0 | 0.0471535 | False | False |
| datadog_agent_remap_datadog_logs_acks | 331.31KiB | 0.54 | 99.41% | 59.7MiB | 3.45MiB | 72.04KiB | 0 | 0.0576955 | 60.03MiB | 4.63MiB | 96.46KiB | 0 | 0.0771836 | False | False |
| http_pipelines_no_grok_blackhole | 26.28KiB | 0.24 | 83.62% | 10.74MiB | 142.21KiB | 2.9KiB | 0 | 0.0129222 | 10.77MiB | 915.71KiB | 18.64KiB | 0 | 0.0830111 | False | False |
| socket_to_socket_blackhole | 51.2KiB | 0.23 | 100.00% | 21.88MiB | 264.79KiB | 5.4KiB | 0 | 0.0118182 | 21.93MiB | 240.59KiB | 4.91KiB | 0 | 0.0107138 | False | False |
| datadog_agent_remap_blackhole | 109.08KiB | 0.18 | 69.81% | 60.86MiB | 4.01MiB | 83.65KiB | 0 | 0.0659439 | 60.97MiB | 3.09MiB | 64.54KiB | 0 | 0.0507416 | False | False |
| syslog_splunk_hec_logs | 25.85KiB | 0.16 | 80.58% | 15.83MiB | 660.9KiB | 13.46KiB | 0 | 0.0407622 | 15.86MiB | 720.13KiB | 14.67KiB | 0 | 0.0443447 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 25.44KiB | 0.1 | 94.88% | 23.81MiB | 547.42KiB | 11.16KiB | 0 | 0.0224465 | 23.84MiB | 330.31KiB | 6.74KiB | 0 | 0.0135301 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 18.12KiB | 0.07 | 53.37% | 23.74MiB | 899.06KiB | 18.28KiB | 0 | 0.0369743 | 23.76MiB | 828.66KiB | 16.86KiB | 0 | 0.0340536 | False | False |
| datadog_agent_remap_blackhole_acks | 25.93KiB | 0.04 | 18.20% | 58.52MiB | 4.43MiB | 92.35KiB | 0 | 0.0757515 | 58.54MiB | 3.08MiB | 64.54KiB | 0 | 0.0526398 | False | False |
| splunk_hec_indexer_ack_blackhole | 9.7KiB | 0.04 | 29.11% | 23.74MiB | 910.96KiB | 18.52KiB | 0 | 0.037465 | 23.75MiB | 896.02KiB | 18.23KiB | 0 | 0.0368359 | False | False |
| datadog_agent_remap_datadog_logs | 21.82KiB | 0.03 | 21.49% | 62.15MiB | 275.21KiB | 5.63KiB | 0 | 0.00432338 | 62.17MiB | 3.83MiB | 79.8KiB | 0 | 0.061587 | False | False |
| enterprise_http_to_http | -574.25B | -0 | 6.16% | 23.85MiB | 251.36KiB | 5.13KiB | 0 | 0.0102918 | 23.85MiB | 251.09KiB | 5.14KiB | 0 | 0.0102808 | False | False |
| file_to_blackhole | -89.44KiB | -0.09 | 71.04% | 95.36MiB | 2.63MiB | 54.55KiB | 0 | 0.0275897 | 95.27MiB | 3.1MiB | 64.47KiB | 0 | 0.0325532 | False | False |
| fluent_elasticsearch | -130.79KiB | -0.16 | 100.00% | 79.47MiB | 53.92KiB | 1.09KiB | 0 | 0.000662379 | 79.35MiB | 1.24MiB | 25.57KiB | 0 | 0.0156545 | False | False |
| http_to_http_json | -47.05KiB | -0.19 | 99.94% | 23.85MiB | 338.8KiB | 6.92KiB | 0 | 0.0138722 | 23.8MiB | 579.85KiB | 11.83KiB | 0 | 0.0237879 | False | False |
| http_pipelines_blackhole | -5.96KiB | -0.36 | 99.09% | 1.63MiB | 47.02KiB | 984.82B | 0 | 0.028186 | 1.62MiB | 101.63KiB | 2.07KiB | 0 | 0.0611429 | False | False |
| http_to_http_noack | -88.18KiB | -0.36 | 99.99% | 23.85MiB | 257.45KiB | 5.27KiB | 0 | 0.0105413 | 23.76MiB | 1.02MiB | 21.2KiB | 0 | 0.0427696 | False | False |
| http_to_http_acks | -296.7KiB | -1.65 | 79.73% | 17.59MiB | 7.83MiB | 163.62KiB | 0 | 0.444901 | 17.3MiB | 7.94MiB | 165.74KiB | 0 | 0.4588 | True | True |
| syslog_log2metric_humio_metrics | -340.78KiB | -2.61 | 100.00% | 12.77MiB | 328.49KiB | 6.7KiB | 0 | 0.0251149 | 12.44MiB | 566.48KiB | 11.53KiB | 0 | 0.0444687 | False | False |
Soak Test Results
Baseline: d9bf36d070a78dd1aac70190577a80a3158dc62b Comparison: f87b3bc3ac02a7019884ad43b089bc2928eedc3c Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| socket_to_socket_blackhole | 389.63KiB | 1.69 | 100.00% | 22.53MiB | 300.4KiB | 6.13KiB | 0 | 0.013017 | 22.91MiB | 465.87KiB | 9.51KiB | 0 | 0.019852 | False | False |
| http_text_to_http_json | 630.01KiB | 1.65 | 100.00% | 37.33MiB | 850.28KiB | 17.36KiB | 0 | 0.022241 | 37.94MiB | 910.32KiB | 18.59KiB | 0 | 0.0234255 | False | False |
| syslog_loki | 109.13KiB | 0.75 | 100.00% | 14.14MiB | 390.29KiB | 7.98KiB | 0 | 0.0269449 | 14.25MiB | 743.52KiB | 15.11KiB | 0 | 0.050947 | False | False |
| syslog_humio_logs | 98.69KiB | 0.59 | 100.00% | 16.21MiB | 112.04KiB | 2.29KiB | 0 | 0.00674972 | 16.3MiB | 111.59KiB | 2.29KiB | 0 | 0.00668311 | False | False |
| syslog_log2metric_splunk_hec_metrics | 78.53KiB | 0.45 | 99.88% | 16.91MiB | 857.17KiB | 17.46KiB | 0 | 0.0494894 | 16.99MiB | 829.06KiB | 16.89KiB | 0 | 0.0476503 | False | False |
| http_pipelines_blackhole_acks | 2.87KiB | 0.23 | 66.83% | 1.22MiB | 117.22KiB | 2.38KiB | 0 | 0.094068 | 1.22MiB | 86.09KiB | 1.75KiB | 0 | 0.0689248 | False | False |
| syslog_regex_logs2metric_ddmetrics | 28.78KiB | 0.23 | 92.18% | 12.26MiB | 594.05KiB | 12.1KiB | 0 | 0.0472941 | 12.29MiB | 538.75KiB | 10.98KiB | 0 | 0.0427939 | False | False |
| splunk_hec_route_s3 | 41.67KiB | 0.23 | 44.79% | 17.99MiB | 2.42MiB | 50.46KiB | 0 | 0.134574 | 18.03MiB | 2.32MiB | 48.63KiB | 0 | 0.128881 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 17.42KiB | 0.07 | 85.68% | 23.82MiB | 477.53KiB | 9.75KiB | 0 | 0.0195729 | 23.84MiB | 333.68KiB | 6.81KiB | 0 | 0.0136671 | False | False |
| http_pipelines_blackhole | 538.15B | 0.03 | 14.00% | 1.64MiB | 64.69KiB | 1.32KiB | 0 | 0.0385444 | 1.64MiB | 130.96KiB | 2.67KiB | 0 | 0.0780022 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 3.62KiB | 0.01 | 12.65% | 23.77MiB | 801.81KiB | 16.32KiB | 0 | 0.0329392 | 23.77MiB | 777.44KiB | 15.83KiB | 0 | 0.0319331 | False | False |
| datadog_agent_remap_datadog_logs_acks | -366.95B | -0 | 0.26% | 61.38MiB | 2.79MiB | 58.34KiB | 0 | 0.045425 | 61.38MiB | 4.4MiB | 91.52KiB | 0 | 0.0716115 | False | False |
| enterprise_http_to_http | -504.3B | -0 | 5.37% | 23.85MiB | 252.0KiB | 5.14KiB | 0 | 0.0103176 | 23.85MiB | 254.23KiB | 5.2KiB | 0 | 0.0104092 | False | False |
| splunk_hec_indexer_ack_blackhole | -8.1KiB | -0.03 | 24.91% | 23.75MiB | 868.31KiB | 17.66KiB | 0 | 0.0356942 | 23.74MiB | 906.36KiB | 18.43KiB | 0 | 0.0372709 | False | False |
| file_to_blackhole | -60.7KiB | -0.06 | 48.30% | 95.34MiB | 2.94MiB | 60.99KiB | 0 | 0.0308513 | 95.28MiB | 3.42MiB | 71.09KiB | 0 | 0.0358802 | False | False |
| http_to_http_json | -28.4KiB | -0.12 | 97.77% | 23.84MiB | 343.54KiB | 7.01KiB | 0 | 0.0140674 | 23.82MiB | 501.47KiB | 10.25KiB | 0 | 0.0205585 | False | False |
| http_to_http_noack | -40.43KiB | -0.17 | 93.19% | 23.82MiB | 594.15KiB | 12.15KiB | 0 | 0.0243543 | 23.78MiB | 909.01KiB | 18.52KiB | 0 | 0.0373221 | False | False |
| fluent_elasticsearch | -161.82KiB | -0.2 | 100.00% | 79.47MiB | 52.77KiB | 1.07KiB | 0 | 0.000648349 | 79.32MiB | 1.57MiB | 32.32KiB | 0 | 0.0197668 | False | False |
| syslog_log2metric_humio_metrics | -36.53KiB | -0.29 | 99.92% | 12.34MiB | 177.4KiB | 3.62KiB | 0 | 0.014031 | 12.31MiB | 501.76KiB | 10.22KiB | 0 | 0.0398018 | False | False |
| datadog_agent_remap_blackhole | -206.11KiB | -0.33 | 96.21% | 60.4MiB | 4.1MiB | 85.42KiB | 0 | 0.0678856 | 60.2MiB | 2.42MiB | 50.5KiB | 0 | 0.0401759 | False | False |
| syslog_splunk_hec_logs | -63.61KiB | -0.38 | 99.86% | 16.26MiB | 659.73KiB | 13.44KiB | 0 | 0.039617 | 16.2MiB | 722.94KiB | 14.72KiB | 0 | 0.0435794 | False | False |
| datadog_agent_remap_datadog_logs | -247.99KiB | -0.4 | 99.30% | 60.19MiB | 1.35MiB | 28.41KiB | 0 | 0.0225036 | 59.94MiB | 4.2MiB | 87.46KiB | 0 | 0.0700303 | False | False |
| datadog_agent_remap_blackhole_acks | -397.12KiB | -0.63 | 99.98% | 61.25MiB | 4.34MiB | 90.44KiB | 0 | 0.0709099 | 60.86MiB | 2.71MiB | 56.55KiB | 0 | 0.0444377 | False | False |
| http_to_http_acks | -194.71KiB | -1.11 | 60.54% | 17.15MiB | 7.74MiB | 161.76KiB | 0 | 0.451081 | 16.96MiB | 7.74MiB | 161.69KiB | 0 | 0.456506 | True | True |
| http_pipelines_no_grok_blackhole | -178.95KiB | -1.65 | 100.00% | 10.61MiB | 158.12KiB | 3.23KiB | 0 | 0.0145484 | 10.44MiB | 986.11KiB | 20.06KiB | 0 | 0.0922524 | False | False |