vector
vector copied to clipboard
feat(pulsar): message keys as config field
Deploy Preview for vector-project canceled.
Name | Link |
---|---|
Latest commit | ba16a27eaf3c0b03fccc7c8e18f748795e43afe8 |
Latest deploy log | https://app.netlify.com/sites/vector-project/deploys/632c788a05cc5e0009afa400 |
Soak Test Results
Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: 2db1721b69d542e4bf1c9a3bf2301c07ce2229aa Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_pipelines_blackhole_acks | 2.84KiB | 0.22 | 67.29% | 1.24MiB | 111.28KiB | 2.26KiB | 0 | 0.0878911 | 1.24MiB | 88.51KiB | 1.8KiB | 0 | 0.0697547 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 16.7KiB | 0.07 | 83.96% | 23.82MiB | 476.8KiB | 9.73KiB | 0 | 0.0195434 | 23.84MiB | 334.94KiB | 6.84KiB | 0 | 0.0137195 | False | False |
enterprise_http_to_http | 2.13KiB | 0.01 | 22.50% | 23.84MiB | 254.75KiB | 5.2KiB | 0 | 0.0104318 | 23.85MiB | 260.48KiB | 5.33KiB | 0 | 0.0106656 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -1.47KiB | -0.01 | 4.47% | 23.74MiB | 906.88KiB | 18.44KiB | 0 | 0.0372987 | 23.74MiB | 915.15KiB | 18.61KiB | 0 | 0.0376408 | False | False |
splunk_hec_indexer_ack_blackhole | -6.57KiB | -0.03 | 19.98% | 23.75MiB | 882.58KiB | 17.95KiB | 0 | 0.0362834 | 23.74MiB | 923.63KiB | 18.78KiB | 0 | 0.0379813 | False | False |
file_to_blackhole | -42.25KiB | -0.04 | 31.09% | 95.34MiB | 3.49MiB | 72.26KiB | 0 | 0.036557 | 95.29MiB | 3.7MiB | 76.99KiB | 0 | 0.038829 | False | False |
socket_to_socket_blackhole | -21.17KiB | -0.09 | 97.00% | 23.18MiB | 121.41KiB | 2.48KiB | 0 | 0.00511429 | 23.16MiB | 461.96KiB | 9.43KiB | 0 | 0.0194771 | False | False |
http_to_http_json | -40.53KiB | -0.17 | 99.86% | 23.85MiB | 341.47KiB | 6.97KiB | 0 | 0.0139809 | 23.81MiB | 519.63KiB | 10.6KiB | 0 | 0.0213111 | False | False |
fluent_elasticsearch | -174.95KiB | -0.21 | 100.00% | 79.47MiB | 53.53KiB | 1.08KiB | 0 | 0.000657593 | 79.3MiB | 1.58MiB | 32.62KiB | 0 | 0.0199804 | False | False |
syslog_regex_logs2metric_ddmetrics | -33.21KiB | -0.26 | 95.89% | 12.51MiB | 602.98KiB | 12.29KiB | 0 | 0.0470521 | 12.48MiB | 522.39KiB | 10.65KiB | 0 | 0.0408693 | False | False |
http_to_http_noack | -95.22KiB | -0.39 | 100.00% | 23.85MiB | 253.62KiB | 5.19KiB | 0 | 0.0103841 | 23.75MiB | 1.05MiB | 21.95KiB | 0 | 0.0443113 | False | False |
datadog_agent_remap_blackhole_acks | -343.39KiB | -0.57 | 99.60% | 59.04MiB | 4.74MiB | 98.76KiB | 0 | 0.0803309 | 58.7MiB | 3.2MiB | 66.95KiB | 0 | 0.0544216 | False | False |
syslog_splunk_hec_logs | -172.73KiB | -1.03 | 100.00% | 16.32MiB | 752.89KiB | 15.32KiB | 0 | 0.0450298 | 16.16MiB | 659.56KiB | 13.45KiB | 0 | 0.0398593 | False | False |
datadog_agent_remap_blackhole | -650.32KiB | -1.05 | 100.00% | 60.29MiB | 3.66MiB | 76.36KiB | 0 | 0.0607056 | 59.66MiB | 3.16MiB | 65.9KiB | 0 | 0.0529442 | False | False |
syslog_log2metric_humio_metrics | -139.58KiB | -1.1 | 100.00% | 12.34MiB | 408.55KiB | 8.34KiB | 0 | 0.0323156 | 12.21MiB | 456.19KiB | 9.29KiB | 0 | 0.0364866 | False | False |
splunk_hec_route_s3 | -234.39KiB | -1.21 | 99.96% | 18.98MiB | 2.26MiB | 47.02KiB | 0 | 0.11898 | 18.75MiB | 2.22MiB | 46.5KiB | 0 | 0.118625 | False | False |
datadog_agent_remap_datadog_logs | -873.52KiB | -1.38 | 100.00% | 61.97MiB | 2.54MiB | 53.22KiB | 0 | 0.0409348 | 61.12MiB | 4.05MiB | 84.31KiB | 0 | 0.0662171 | False | False |
syslog_humio_logs | -250.91KiB | -1.47 | 100.00% | 16.67MiB | 129.33KiB | 2.64KiB | 0 | 0.00757718 | 16.42MiB | 121.32KiB | 2.48KiB | 0 | 0.00721355 | False | False |
datadog_agent_remap_datadog_logs_acks | -984.75KiB | -1.54 | 100.00% | 62.32MiB | 3.41MiB | 71.16KiB | 0 | 0.054662 | 61.36MiB | 4.36MiB | 90.68KiB | 0 | 0.0709805 | False | False |
http_to_http_acks | -286.55KiB | -1.59 | 78.37% | 17.59MiB | 8.02MiB | 167.61KiB | 0 | 0.455753 | 17.31MiB | 7.66MiB | 160.04KiB | 0 | 0.442664 | True | True |
http_pipelines_no_grok_blackhole | -181.01KiB | -1.62 | 100.00% | 10.88MiB | 50.87KiB | 1.04KiB | 0 | 0.00456505 | 10.7MiB | 1.06MiB | 22.13KiB | 0 | 0.0992963 | False | False |
syslog_log2metric_splunk_hec_metrics | -353.33KiB | -1.97 | 100.00% | 17.52MiB | 689.79KiB | 14.06KiB | 0 | 0.0384426 | 17.17MiB | 891.56KiB | 18.15KiB | 0 | 0.0506856 | False | False |
http_pipelines_blackhole | -41.1KiB | -2.33 | 100.00% | 1.72MiB | 10.53KiB | 220.59B | 0 | 0.00597423 | 1.68MiB | 118.61KiB | 2.42KiB | 0 | 0.0688938 | False | False |
syslog_loki | -409.0KiB | -2.74 | 100.00% | 14.58MiB | 422.06KiB | 8.63KiB | 0 | 0.0282659 | 14.18MiB | 709.07KiB | 14.41KiB | 0 | 0.0488255 | False | False |
http_text_to_http_json | -2.06MiB | -5.2 | 100.00% | 39.6MiB | 748.64KiB | 15.28KiB | 0 | 0.0184574 | 37.54MiB | 816.11KiB | 16.66KiB | 0 | 0.0212255 | False | False |
Soak Test Results
Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: 1884b926cacde40a73e7087eb93ee18eb51bdfc9 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_to_http_acks | 201.51KiB | 1.14 | 61.57% | 17.25MiB | 8.04MiB | 168.22KiB | 0 | 0.466202 | 17.45MiB | 7.63MiB | 159.21KiB | 0 | 0.437266 | True | True |
syslog_log2metric_humio_metrics | 15.78KiB | 0.13 | 71.02% | 12.22MiB | 476.44KiB | 9.71KiB | 0 | 0.0380729 | 12.23MiB | 555.59KiB | 11.31KiB | 0 | 0.0443417 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 26.02KiB | 0.11 | 95.65% | 23.81MiB | 531.31KiB | 10.84KiB | 0 | 0.0217858 | 23.84MiB | 341.4KiB | 6.97KiB | 0 | 0.0139836 | False | False |
splunk_hec_indexer_ack_blackhole | 16.67KiB | 0.07 | 44.89% | 23.73MiB | 998.41KiB | 20.3KiB | 0 | 0.0410822 | 23.74MiB | 945.55KiB | 19.23KiB | 0 | 0.0388804 | False | False |
splunk_hec_to_splunk_hec_logs_acks | 1.77KiB | 0.01 | 5.71% | 23.75MiB | 854.37KiB | 17.38KiB | 0 | 0.0351215 | 23.75MiB | 859.57KiB | 17.48KiB | 0 | 0.0353325 | False | False |
enterprise_http_to_http | -778.3B | -0 | 8.23% | 23.84MiB | 252.2KiB | 5.15KiB | 0 | 0.0103267 | 23.84MiB | 257.05KiB | 5.26KiB | 0 | 0.0105259 | False | False |
file_to_blackhole | -93.41KiB | -0.1 | 65.03% | 95.34MiB | 3.09MiB | 64.14KiB | 0 | 0.0324473 | 95.25MiB | 3.68MiB | 76.56KiB | 0 | 0.038671 | False | False |
http_to_http_json | -40.2KiB | -0.16 | 99.81% | 23.85MiB | 329.64KiB | 6.73KiB | 0 | 0.0134957 | 23.81MiB | 539.54KiB | 11.02KiB | 0 | 0.0221253 | False | False |
fluent_elasticsearch | -162.72KiB | -0.2 | 100.00% | 79.47MiB | 54.94KiB | 1.11KiB | 0 | 0.000674912 | 79.31MiB | 1.53MiB | 31.46KiB | 0 | 0.0192627 | False | False |
http_pipelines_blackhole_acks | -3.34KiB | -0.26 | 79.99% | 1.23MiB | 107.85KiB | 2.2KiB | 0 | 0.0854625 | 1.23MiB | 68.84KiB | 1.4KiB | 0 | 0.054694 | False | False |
socket_to_socket_blackhole | -69.24KiB | -0.3 | 100.00% | 22.92MiB | 190.76KiB | 3.89KiB | 0 | 0.00812733 | 22.85MiB | 394.1KiB | 8.05KiB | 0 | 0.0168404 | False | False |
syslog_regex_logs2metric_ddmetrics | -48.13KiB | -0.38 | 99.23% | 12.32MiB | 633.41KiB | 12.91KiB | 0 | 0.0501957 | 12.27MiB | 618.51KiB | 12.6KiB | 0 | 0.0492026 | False | False |
http_to_http_noack | -101.08KiB | -0.41 | 100.00% | 23.84MiB | 409.74KiB | 8.37KiB | 0 | 0.0167833 | 23.74MiB | 1.12MiB | 23.25KiB | 0 | 0.0469744 | False | False |
datadog_agent_remap_blackhole | -290.81KiB | -0.47 | 99.23% | 60.77MiB | 4.05MiB | 84.33KiB | 0 | 0.0665791 | 60.49MiB | 3.31MiB | 69.08KiB | 0 | 0.0547543 | False | False |
syslog_splunk_hec_logs | -157.06KiB | -0.94 | 100.00% | 16.34MiB | 703.33KiB | 14.32KiB | 0 | 0.0420229 | 16.19MiB | 622.17KiB | 12.69KiB | 0 | 0.0375261 | False | False |
http_pipelines_blackhole | -17.95KiB | -1.09 | 100.00% | 1.61MiB | 47.24KiB | 988.29B | 0 | 0.0285776 | 1.6MiB | 118.46KiB | 2.41KiB | 0 | 0.0724472 | False | False |
datadog_agent_remap_blackhole_acks | -709.7KiB | -1.16 | 100.00% | 59.78MiB | 4.84MiB | 100.84KiB | 0 | 0.080938 | 59.09MiB | 3.75MiB | 78.51KiB | 0 | 0.0634484 | False | False |
syslog_humio_logs | -210.19KiB | -1.25 | 100.00% | 16.39MiB | 476.9KiB | 9.74KiB | 0 | 0.0284169 | 16.18MiB | 437.42KiB | 8.96KiB | 0 | 0.0263952 | False | False |
syslog_log2metric_splunk_hec_metrics | -330.46KiB | -1.79 | 100.00% | 18.02MiB | 639.55KiB | 13.03KiB | 0 | 0.0346604 | 17.69MiB | 691.64KiB | 14.09KiB | 0 | 0.0381668 | False | False |
datadog_agent_remap_datadog_logs_acks | -1.11MiB | -1.82 | 100.00% | 60.96MiB | 3.43MiB | 71.7KiB | 0 | 0.0562866 | 59.85MiB | 4.44MiB | 92.41KiB | 0 | 0.0741664 | False | False |
datadog_agent_remap_datadog_logs | -1.19MiB | -1.93 | 100.00% | 61.74MiB | 1.38MiB | 28.86KiB | 0 | 0.022289 | 60.55MiB | 3.96MiB | 82.5KiB | 0 | 0.0653809 | False | False |
syslog_loki | -533.24KiB | -3.62 | 100.00% | 14.4MiB | 410.02KiB | 8.39KiB | 0 | 0.0277992 | 13.88MiB | 705.16KiB | 14.33KiB | 0 | 0.0496028 | False | False |
http_text_to_http_json | -2.06MiB | -5.21 | 100.00% | 39.52MiB | 813.94KiB | 16.61KiB | 0 | 0.0201082 | 37.46MiB | 795.21KiB | 16.23KiB | 0 | 0.0207247 | False | False |
splunk_hec_route_s3 | -1015.68KiB | -5.63 | 100.00% | 17.62MiB | 2.58MiB | 53.82KiB | 0 | 0.146661 | 16.62MiB | 2.64MiB | 55.16KiB | 0 | 0.158619 | False | False |
http_pipelines_no_grok_blackhole | -664.67KiB | -6.27 | 100.00% | 10.35MiB | 771.07KiB | 15.74KiB | 0 | 0.0727366 | 9.7MiB | 1.36MiB | 28.33KiB | 0 | 0.140221 | False | False |
Soak Test Results
Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: 5de4d5556dbf26f46e2966fafd3e4acb134077d3 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_to_http_acks | 377.12KiB | 2.13 | 89.14% | 17.33MiB | 8.11MiB | 169.6KiB | 0 | 0.4681 | 17.7MiB | 7.79MiB | 162.66KiB | 0 | 0.439878 | True | True |
syslog_log2metric_humio_metrics | 31.0KiB | 0.24 | 97.62% | 12.61MiB | 368.16KiB | 7.52KiB | 0 | 0.0284953 | 12.64MiB | 563.46KiB | 11.47KiB | 0 | 0.0435072 | False | False |
splunk_hec_indexer_ack_blackhole | 39.58KiB | 0.16 | 90.41% | 23.74MiB | 903.79KiB | 18.38KiB | 0 | 0.0371693 | 23.78MiB | 740.38KiB | 15.08KiB | 0 | 0.0303994 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 7.35KiB | 0.03 | 52.34% | 23.83MiB | 379.68KiB | 7.76KiB | 0 | 0.0155555 | 23.84MiB | 333.94KiB | 6.82KiB | 0 | 0.0136773 | False | False |
enterprise_http_to_http | -1.04KiB | -0 | 11.43% | 23.85MiB | 247.59KiB | 5.05KiB | 0 | 0.0101374 | 23.85MiB | 251.03KiB | 5.13KiB | 0 | 0.0102784 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -4.62KiB | -0.02 | 15.48% | 23.76MiB | 807.11KiB | 16.42KiB | 0 | 0.0331646 | 23.76MiB | 835.36KiB | 17.0KiB | 0 | 0.0343317 | False | False |
file_to_blackhole | -48.04KiB | -0.05 | 32.07% | 95.33MiB | 3.71MiB | 76.87KiB | 0 | 0.03889 | 95.29MiB | 4.19MiB | 87.14KiB | 0 | 0.043924 | False | False |
http_to_http_json | -37.35KiB | -0.15 | 99.64% | 23.85MiB | 344.15KiB | 7.03KiB | 0 | 0.0140908 | 23.81MiB | 526.29KiB | 10.75KiB | 0 | 0.0215811 | False | False |
fluent_elasticsearch | -180.57KiB | -0.22 | 100.00% | 79.47MiB | 54.15KiB | 1.1KiB | 0 | 0.00066524 | 79.3MiB | 1.62MiB | 33.3KiB | 0 | 0.0204006 | False | False |
datadog_agent_remap_blackhole_acks | -236.97KiB | -0.37 | 98.54% | 62.37MiB | 3.91MiB | 81.41KiB | 0 | 0.0626302 | 62.14MiB | 2.52MiB | 52.66KiB | 0 | 0.0405131 | False | False |
http_to_http_noack | -96.06KiB | -0.39 | 99.97% | 23.83MiB | 519.44KiB | 10.62KiB | 0 | 0.0212836 | 23.73MiB | 1.15MiB | 24.0KiB | 0 | 0.0484928 | False | False |
http_pipelines_blackhole_acks | -5.66KiB | -0.45 | 95.81% | 1.22MiB | 108.41KiB | 2.21KiB | 0 | 0.0866845 | 1.22MiB | 83.09KiB | 1.69KiB | 0 | 0.0667362 | False | False |
syslog_splunk_hec_logs | -114.08KiB | -0.71 | 100.00% | 15.67MiB | 731.55KiB | 14.88KiB | 0 | 0.0455836 | 15.56MiB | 605.54KiB | 12.36KiB | 0 | 0.038002 | False | False |
syslog_regex_logs2metric_ddmetrics | -92.6KiB | -0.72 | 100.00% | 12.63MiB | 496.11KiB | 10.12KiB | 0 | 0.0383534 | 12.54MiB | 569.07KiB | 11.59KiB | 0 | 0.0443111 | False | False |
socket_to_socket_blackhole | -176.2KiB | -0.74 | 100.00% | 23.17MiB | 134.32KiB | 2.74KiB | 0 | 0.00565905 | 23.0MiB | 482.98KiB | 9.86KiB | 0 | 0.0205009 | False | False |
syslog_log2metric_splunk_hec_metrics | -175.2KiB | -0.99 | 100.00% | 17.36MiB | 660.59KiB | 13.45KiB | 0 | 0.0371463 | 17.19MiB | 720.83KiB | 14.67KiB | 0 | 0.040937 | False | False |
http_pipelines_blackhole | -21.68KiB | -1.25 | 100.00% | 1.69MiB | 12.66KiB | 264.76B | 0 | 0.00729118 | 1.67MiB | 126.03KiB | 2.57KiB | 0 | 0.0735242 | False | False |
datadog_agent_remap_datadog_logs | -873.49KiB | -1.37 | 100.00% | 62.32MiB | 1.29MiB | 26.94KiB | 0 | 0.0206198 | 61.47MiB | 4.15MiB | 86.37KiB | 0 | 0.0674579 | False | False |
datadog_agent_remap_blackhole | -855.06KiB | -1.39 | 100.00% | 60.07MiB | 4.16MiB | 86.64KiB | 0 | 0.0692442 | 59.24MiB | 3.06MiB | 63.92KiB | 0 | 0.0517209 | False | False |
syslog_humio_logs | -254.49KiB | -1.49 | 100.00% | 16.63MiB | 123.63KiB | 2.52KiB | 0 | 0.00725923 | 16.38MiB | 126.48KiB | 2.59KiB | 0 | 0.0075397 | False | False |
datadog_agent_remap_datadog_logs_acks | -946.41KiB | -1.51 | 100.00% | 61.27MiB | 3.45MiB | 71.96KiB | 0 | 0.0562252 | 60.35MiB | 4.32MiB | 89.83KiB | 0 | 0.0714977 | False | False |
http_pipelines_no_grok_blackhole | -184.77KiB | -1.66 | 100.00% | 10.88MiB | 57.53KiB | 1.17KiB | 0 | 0.00516074 | 10.7MiB | 1.06MiB | 22.15KiB | 0 | 0.0993959 | False | False |
splunk_hec_route_s3 | -324.63KiB | -1.69 | 100.00% | 18.73MiB | 2.26MiB | 47.03KiB | 0 | 0.120485 | 18.42MiB | 2.22MiB | 46.46KiB | 0 | 0.120633 | False | False |
syslog_loki | -262.08KiB | -1.95 | 100.00% | 13.14MiB | 927.81KiB | 19.0KiB | 0 | 0.0689576 | 12.88MiB | 1.17MiB | 24.44KiB | 0 | 0.0911303 | False | False |
http_text_to_http_json | -2.14MiB | -5.48 | 100.00% | 39.01MiB | 979.02KiB | 19.98KiB | 0 | 0.0245061 | 36.87MiB | 830.09KiB | 16.95KiB | 0 | 0.021984 | False | False |
Soak Test Results
Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: d146ad5976d323a8a6db7881dde879adbe1ab3ee Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_pipelines_blackhole_acks | 13.67KiB | 1.1 | 100.00% | 1.22MiB | 109.71KiB | 2.23KiB | 0 | 0.0881505 | 1.23MiB | 37.61KiB | 785.87B | 0 | 0.0298883 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 11.36KiB | 0.05 | 69.05% | 23.83MiB | 434.95KiB | 8.88KiB | 0 | 0.0178224 | 23.84MiB | 332.46KiB | 6.79KiB | 0 | 0.0136166 | False | False |
splunk_hec_indexer_ack_blackhole | 10.08KiB | 0.04 | 31.75% | 23.75MiB | 872.86KiB | 17.75KiB | 0 | 0.035882 | 23.76MiB | 839.21KiB | 17.08KiB | 0 | 0.0344847 | False | False |
enterprise_http_to_http | 6.61B | 0 | 0.07% | 23.85MiB | 249.41KiB | 5.09KiB | 0 | 0.0102123 | 23.85MiB | 251.67KiB | 5.15KiB | 0 | 0.0103047 | False | False |
file_to_blackhole | -61.09KiB | -0.06 | 49.46% | 95.35MiB | 3.01MiB | 62.42KiB | 0 | 0.0315752 | 95.29MiB | 3.23MiB | 67.2KiB | 0 | 0.0339227 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -15.9KiB | -0.07 | 49.07% | 23.76MiB | 805.93KiB | 16.4KiB | 0 | 0.0331131 | 23.75MiB | 867.87KiB | 17.65KiB | 0 | 0.0356814 | False | False |
http_to_http_json | -38.9KiB | -0.16 | 99.74% | 23.85MiB | 329.79KiB | 6.73KiB | 0 | 0.0135025 | 23.81MiB | 539.78KiB | 11.02KiB | 0 | 0.0221353 | False | False |
fluent_elasticsearch | -212.68KiB | -0.26 | 100.00% | 79.47MiB | 53.18KiB | 1.08KiB | 0 | 0.000653314 | 79.27MiB | 1.82MiB | 37.55KiB | 0 | 0.023017 | False | False |
http_to_http_noack | -95.13KiB | -0.39 | 99.99% | 23.84MiB | 412.06KiB | 8.43KiB | 0 | 0.0168774 | 23.74MiB | 1.09MiB | 22.79KiB | 0 | 0.0460266 | False | False |
datadog_agent_remap_blackhole_acks | -402.75KiB | -0.63 | 100.00% | 62.32MiB | 4.26MiB | 88.69KiB | 0 | 0.068324 | 61.93MiB | 1.74MiB | 36.54KiB | 0 | 0.0281514 | False | False |
splunk_hec_route_s3 | -134.54KiB | -0.72 | 95.28% | 18.16MiB | 2.32MiB | 48.35KiB | 0 | 0.127744 | 18.03MiB | 2.27MiB | 47.51KiB | 0 | 0.125961 | False | False |
syslog_regex_logs2metric_ddmetrics | -95.17KiB | -0.75 | 100.00% | 12.39MiB | 556.32KiB | 11.33KiB | 0 | 0.043836 | 12.3MiB | 524.44KiB | 10.69KiB | 0 | 0.0416367 | False | False |
socket_to_socket_blackhole | -239.49KiB | -1.04 | 100.00% | 22.51MiB | 725.46KiB | 14.81KiB | 0 | 0.0314596 | 22.28MiB | 947.63KiB | 19.35KiB | 0 | 0.0415251 | False | False |
http_to_http_acks | -214.76KiB | -1.19 | 64.51% | 17.62MiB | 8.15MiB | 170.36KiB | 0 | 0.462522 | 17.41MiB | 7.55MiB | 157.68KiB | 0 | 0.433869 | True | True |
syslog_splunk_hec_logs | -208.7KiB | -1.23 | 100.00% | 16.62MiB | 782.41KiB | 15.92KiB | 0 | 0.0459699 | 16.41MiB | 641.47KiB | 13.07KiB | 0 | 0.038157 | False | False |
datadog_agent_remap_datadog_logs | -971.21KiB | -1.54 | 100.00% | 61.4MiB | 823.98KiB | 16.87KiB | 0 | 0.0131035 | 60.45MiB | 4.07MiB | 84.8KiB | 0 | 0.0673503 | False | False |
syslog_log2metric_humio_metrics | -200.5KiB | -1.58 | 100.00% | 12.43MiB | 405.37KiB | 8.28KiB | 0 | 0.0318477 | 12.23MiB | 626.32KiB | 12.74KiB | 0 | 0.0499943 | False | False |
syslog_humio_logs | -265.44KiB | -1.59 | 100.00% | 16.29MiB | 545.67KiB | 11.14KiB | 0 | 0.0327024 | 16.03MiB | 592.5KiB | 12.13KiB | 0 | 0.0360834 | False | False |
datadog_agent_remap_datadog_logs_acks | -1015.02KiB | -1.61 | 100.00% | 61.58MiB | 2.94MiB | 61.48KiB | 0 | 0.0477358 | 60.59MiB | 4.27MiB | 88.9KiB | 0 | 0.0704746 | False | False |
http_pipelines_no_grok_blackhole | -182.0KiB | -1.64 | 100.00% | 10.81MiB | 113.53KiB | 2.32KiB | 0 | 0.0102498 | 10.64MiB | 1.03MiB | 21.53KiB | 0 | 0.0971789 | False | False |
datadog_agent_remap_blackhole | -1.18MiB | -2.06 | 100.00% | 57.15MiB | 4.05MiB | 84.48KiB | 0 | 0.070896 | 55.97MiB | 3.2MiB | 66.79KiB | 0 | 0.0571918 | False | False |
http_pipelines_blackhole | -40.63KiB | -2.31 | 100.00% | 1.72MiB | 11.5KiB | 240.69B | 0 | 0.00653673 | 1.68MiB | 115.09KiB | 2.35KiB | 0 | 0.0669587 | False | False |
syslog_log2metric_splunk_hec_metrics | -513.97KiB | -2.88 | 100.00% | 17.42MiB | 697.7KiB | 14.22KiB | 0 | 0.0391041 | 16.92MiB | 796.81KiB | 16.22KiB | 0 | 0.0459837 | False | False |
syslog_loki | -558.36KiB | -3.78 | 100.00% | 14.43MiB | 324.42KiB | 6.65KiB | 0 | 0.021958 | 13.88MiB | 737.42KiB | 14.99KiB | 0 | 0.051873 | False | False |
http_text_to_http_json | -2.05MiB | -5.2 | 100.00% | 39.44MiB | 856.52KiB | 17.48KiB | 0 | 0.0212055 | 37.39MiB | 809.11KiB | 16.52KiB | 0 | 0.021131 | False | False |
Soak Test Results
Baseline: d8ca608861ee14e572a3a41cc49728678b9b7c91 Comparison: b3d5010970b97286db6c24aae57660276452e438 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
syslog_splunk_hec_logs | 734.42KiB | 4.7 | 100.00% | 15.25MiB | 1.14MiB | 23.75KiB | 0 | 0.0747066 | 15.97MiB | 1.25MiB | 26.05KiB | 0 | 0.0782018 | False | False |
socket_to_socket_blackhole | 362.15KiB | 1.59 | 100.00% | 22.31MiB | 586.12KiB | 11.96KiB | 0 | 0.0256501 | 22.66MiB | 451.91KiB | 9.23KiB | 0 | 0.019468 | False | False |
syslog_log2metric_humio_metrics | 120.88KiB | 0.95 | 100.00% | 12.37MiB | 281.95KiB | 5.76KiB | 0 | 0.0222536 | 12.49MiB | 492.83KiB | 10.03KiB | 0 | 0.0385309 | False | False |
syslog_loki | 133.56KiB | 0.91 | 100.00% | 14.32MiB | 472.86KiB | 9.68KiB | 0 | 0.0322423 | 14.45MiB | 729.67KiB | 14.83KiB | 0 | 0.0493043 | False | False |
http_pipelines_blackhole_acks | 8.62KiB | 0.71 | 99.62% | 1.18MiB | 119.41KiB | 2.43KiB | 0 | 0.0986706 | 1.19MiB | 84.23KiB | 1.72KiB | 0 | 0.069114 | False | False |
http_to_http_acks | 124.14KiB | 0.71 | 40.34% | 17.12MiB | 7.91MiB | 165.31KiB | 0 | 0.461806 | 17.24MiB | 7.98MiB | 166.34KiB | 0 | 0.462472 | True | True |
syslog_regex_logs2metric_ddmetrics | 84.75KiB | 0.66 | 100.00% | 12.46MiB | 580.72KiB | 11.83KiB | 0 | 0.0455115 | 12.54MiB | 564.09KiB | 11.5KiB | 0 | 0.0439164 | False | False |
syslog_humio_logs | 71.98KiB | 0.43 | 100.00% | 16.41MiB | 156.63KiB | 3.2KiB | 0 | 0.00931655 | 16.49MiB | 147.89KiB | 3.03KiB | 0 | 0.00875876 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 21.82KiB | 0.09 | 92.06% | 23.82MiB | 504.3KiB | 10.29KiB | 0 | 0.0206727 | 23.84MiB | 341.96KiB | 6.98KiB | 0 | 0.0140056 | False | False |
splunk_hec_route_s3 | 13.94KiB | 0.08 | 15.76% | 18.05MiB | 2.39MiB | 49.78KiB | 0 | 0.132476 | 18.06MiB | 2.36MiB | 49.4KiB | 0 | 0.130788 | False | False |
syslog_log2metric_splunk_hec_metrics | 7.47KiB | 0.04 | 25.42% | 17.33MiB | 791.1KiB | 16.13KiB | 0 | 0.0445636 | 17.34MiB | 808.78KiB | 16.48KiB | 0 | 0.0455402 | False | False |
splunk_hec_to_splunk_hec_logs_acks | 2.26KiB | 0.01 | 7.03% | 23.74MiB | 892.49KiB | 18.15KiB | 0 | 0.0367025 | 23.74MiB | 891.47KiB | 18.13KiB | 0 | 0.0366573 | False | False |
enterprise_http_to_http | -2.68KiB | -0.01 | 28.53% | 23.85MiB | 250.18KiB | 5.11KiB | 0 | 0.010243 | 23.84MiB | 256.66KiB | 5.25KiB | 0 | 0.0105095 | False | False |
splunk_hec_indexer_ack_blackhole | -5.83KiB | -0.02 | 18.59% | 23.76MiB | 851.68KiB | 17.33KiB | 0 | 0.0350051 | 23.75MiB | 873.2KiB | 17.76KiB | 0 | 0.0358983 | False | False |
file_to_blackhole | -100.11KiB | -0.1 | 64.39% | 95.34MiB | 3.38MiB | 69.97KiB | 0 | 0.0353973 | 95.24MiB | 3.99MiB | 82.89KiB | 0 | 0.0418876 | False | False |
http_to_http_json | -31.68KiB | -0.13 | 98.85% | 23.84MiB | 343.7KiB | 7.02KiB | 0 | 0.0140737 | 23.81MiB | 508.51KiB | 10.39KiB | 0 | 0.0208491 | False | False |
fluent_elasticsearch | -164.11KiB | -0.2 | 100.00% | 79.47MiB | 54.37KiB | 1.1KiB | 0 | 0.0006679 | 79.31MiB | 1.54MiB | 31.69KiB | 0 | 0.0193977 | False | False |
http_to_http_noack | -77.3KiB | -0.32 | 99.86% | 23.83MiB | 522.0KiB | 10.66KiB | 0 | 0.0213882 | 23.75MiB | 1.04MiB | 21.69KiB | 0 | 0.0437763 | False | False |
datadog_agent_remap_datadog_logs_acks | -654.88KiB | -1.03 | 100.00% | 61.88MiB | 2.9MiB | 60.67KiB | 0 | 0.0468576 | 61.24MiB | 4.36MiB | 90.77KiB | 0 | 0.0711862 | False | False |
http_pipelines_blackhole | -19.39KiB | -1.12 | 100.00% | 1.69MiB | 14.35KiB | 300.52B | 0 | 0.00828462 | 1.67MiB | 115.91KiB | 2.36KiB | 0 | 0.0676591 | False | False |
datadog_agent_remap_blackhole_acks | -829.67KiB | -1.33 | 100.00% | 61.06MiB | 3.82MiB | 79.65KiB | 0 | 0.0625892 | 60.25MiB | 2.85MiB | 59.7KiB | 0 | 0.0473692 | False | False |
datadog_agent_remap_datadog_logs | -1007.05KiB | -1.58 | 100.00% | 62.42MiB | 748.17KiB | 15.32KiB | 0 | 0.0117019 | 61.44MiB | 4.3MiB | 89.5KiB | 0 | 0.0699537 | False | False |
datadog_agent_remap_blackhole | -1.03MiB | -1.76 | 100.00% | 58.68MiB | 4.3MiB | 89.65KiB | 0 | 0.0733091 | 57.65MiB | 3.76MiB | 78.52KiB | 0 | 0.0652841 | False | False |
http_pipelines_no_grok_blackhole | -229.66KiB | -2.11 | 100.00% | 10.65MiB | 77.58KiB | 1.58KiB | 0 | 0.00711313 | 10.42MiB | 1.01MiB | 21.01KiB | 0 | 0.0967673 | False | False |
http_text_to_http_json | -1.49MiB | -3.83 | 100.00% | 38.96MiB | 1.22MiB | 25.54KiB | 0 | 0.0313566 | 37.47MiB | 814.04KiB | 16.62KiB | 0 | 0.0212112 | False | False |
I'm discovering cue scripts, thanks @fuchsnj
Soak Test Results
Baseline: 248583fe27dfb3c1393c80580af5c3340a89cd3f Comparison: cd5b3112cacad16c2632efd2dc38608865dc9c41 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
socket_to_socket_blackhole | 640.32KiB | 2.89 | 100.00% | 21.66MiB | 747.23KiB | 15.25KiB | 0 | 0.03369 | 22.28MiB | 672.42KiB | 13.73KiB | 0 | 0.0294661 | False | False |
syslog_log2metric_humio_metrics | 248.42KiB | 1.98 | 100.00% | 12.25MiB | 579.48KiB | 11.83KiB | 0 | 0.0461997 | 12.49MiB | 603.89KiB | 12.3KiB | 0 | 0.0472109 | False | False |
syslog_splunk_hec_logs | 214.23KiB | 1.3 | 100.00% | 16.11MiB | 654.6KiB | 13.34KiB | 0 | 0.0396714 | 16.32MiB | 588.67KiB | 12.0KiB | 0 | 0.0352187 | False | False |
syslog_humio_logs | 104.61KiB | 0.62 | 100.00% | 16.36MiB | 256.85KiB | 5.24KiB | 0 | 0.0153296 | 16.46MiB | 192.7KiB | 3.95KiB | 0 | 0.0114299 | False | False |
syslog_regex_logs2metric_ddmetrics | 73.66KiB | 0.58 | 100.00% | 12.49MiB | 587.63KiB | 11.97KiB | 0 | 0.0459319 | 12.56MiB | 516.11KiB | 10.52KiB | 0 | 0.0401104 | False | False |
syslog_loki | 70.58KiB | 0.48 | 100.00% | 14.43MiB | 441.16KiB | 9.03KiB | 0 | 0.0298403 | 14.5MiB | 718.38KiB | 14.6KiB | 0 | 0.0483613 | False | False |
http_pipelines_blackhole_acks | 4.45KiB | 0.36 | 90.64% | 1.22MiB | 99.56KiB | 2.03KiB | 0 | 0.079622 | 1.23MiB | 84.0KiB | 1.71KiB | 0 | 0.06694 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 12.42KiB | 0.05 | 73.33% | 23.83MiB | 432.72KiB | 8.84KiB | 0 | 0.0177324 | 23.84MiB | 335.37KiB | 6.85KiB | 0 | 0.0137364 | False | False |
enterprise_http_to_http | -1.61KiB | -0.01 | 17.43% | 23.85MiB | 248.82KiB | 5.08KiB | 0 | 0.0101872 | 23.85MiB | 257.23KiB | 5.26KiB | 0 | 0.0105321 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -5.0KiB | -0.02 | 15.66% | 23.75MiB | 870.39KiB | 17.7KiB | 0 | 0.0357832 | 23.74MiB | 887.97KiB | 18.06KiB | 0 | 0.0365136 | False | False |
file_to_blackhole | -50.94KiB | -0.05 | 46.28% | 95.35MiB | 2.73MiB | 56.63KiB | 0 | 0.0286449 | 95.3MiB | 2.89MiB | 60.08KiB | 0 | 0.0302927 | False | False |
splunk_hec_indexer_ack_blackhole | -23.95KiB | -0.1 | 63.99% | 23.76MiB | 858.86KiB | 17.48KiB | 0 | 0.0352868 | 23.74MiB | 957.68KiB | 19.48KiB | 0 | 0.0393856 | False | False |
http_to_http_json | -26.65KiB | -0.11 | 97.61% | 23.85MiB | 334.18KiB | 6.82KiB | 0 | 0.0136822 | 23.82MiB | 470.45KiB | 9.62KiB | 0 | 0.0192825 | False | False |
syslog_log2metric_splunk_hec_metrics | -19.49KiB | -0.11 | 52.67% | 17.19MiB | 910.11KiB | 18.56KiB | 0 | 0.0516841 | 17.17MiB | 975.18KiB | 19.86KiB | 0 | 0.0554409 | False | False |
http_to_http_noack | -68.54KiB | -0.28 | 99.96% | 23.85MiB | 261.66KiB | 5.35KiB | 0 | 0.0107133 | 23.78MiB | 917.96KiB | 18.71KiB | 0 | 0.0376901 | False | False |
fluent_elasticsearch | -223.93KiB | -0.28 | 100.00% | 79.47MiB | 52.7KiB | 1.06KiB | 0 | 0.000647437 | 79.25MiB | 2.5MiB | 51.39KiB | 0 | 0.0315038 | False | False |
datadog_agent_remap_blackhole_acks | -305.66KiB | -0.51 | 96.47% | 57.97MiB | 5.3MiB | 110.41KiB | 0 | 0.091437 | 57.67MiB | 4.52MiB | 94.32KiB | 0 | 0.0782681 | False | False |
datadog_agent_remap_datadog_logs | -404.33KiB | -0.66 | 100.00% | 60.23MiB | 1.81MiB | 37.91KiB | 0 | 0.0300135 | 59.84MiB | 4.19MiB | 87.26KiB | 0 | 0.069981 | False | False |
datadog_agent_remap_datadog_logs_acks | -500.19KiB | -0.8 | 100.00% | 61.16MiB | 3.95MiB | 82.35KiB | 0 | 0.0645637 | 60.67MiB | 4.37MiB | 90.96KiB | 0 | 0.0720085 | False | False |
http_to_http_acks | -166.06KiB | -0.94 | 51.43% | 17.24MiB | 8.04MiB | 168.12KiB | 0 | 0.466418 | 17.08MiB | 8.09MiB | 168.74KiB | 0 | 0.473569 | True | True |
http_pipelines_blackhole | -19.7KiB | -1.14 | 100.00% | 1.69MiB | 16.35KiB | 342.34B | 0 | 0.00944109 | 1.67MiB | 119.26KiB | 2.43KiB | 0 | 0.0696432 | False | False |
splunk_hec_route_s3 | -200.51KiB | -1.15 | 99.15% | 17.06MiB | 2.6MiB | 54.16KiB | 0 | 0.152384 | 16.86MiB | 2.56MiB | 53.62KiB | 0 | 0.152018 | False | False |
http_pipelines_no_grok_blackhole | -140.19KiB | -1.33 | 100.00% | 10.32MiB | 292.93KiB | 5.98KiB | 0 | 0.0277046 | 10.19MiB | 1014.36KiB | 20.63KiB | 0 | 0.0972266 | False | False |
datadog_agent_remap_blackhole | -1.13MiB | -1.93 | 100.00% | 58.86MiB | 3.88MiB | 80.93KiB | 0 | 0.0659498 | 57.72MiB | 3.16MiB | 65.96KiB | 0 | 0.05477 | False | False |
http_text_to_http_json | -1.38MiB | -3.56 | 100.00% | 38.72MiB | 1.24MiB | 26.01KiB | 0 | 0.0321284 | 37.34MiB | 788.4KiB | 16.1KiB | 0 | 0.0206143 | False | False |
Soak Test Results
Baseline: 966cd18412bcb0d86a186734690a255758421f0a Comparison: ba16a27eaf3c0b03fccc7c8e18f748795e43afe8 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
syslog_regex_logs2metric_ddmetrics | 493.84KiB | 4.27 | 100.00% | 11.3MiB | 788.78KiB | 16.07KiB | 0 | 0.0681234 | 11.79MiB | 703.62KiB | 14.34KiB | 0 | 0.0582822 | False | False |
syslog_loki | 459.46KiB | 3.19 | 100.00% | 14.07MiB | 423.09KiB | 8.67KiB | 0 | 0.0293617 | 14.52MiB | 723.02KiB | 14.7KiB | 0 | 0.0486262 | False | False |
datadog_agent_remap_blackhole_acks | 1.77MiB | 2.99 | 100.00% | 59.02MiB | 5.84MiB | 121.53KiB | 0 | 0.0988985 | 60.79MiB | 3.58MiB | 74.72KiB | 0 | 0.0588189 | False | False |
http_to_http_acks | 392.61KiB | 2.23 | 90.29% | 17.17MiB | 8.09MiB | 169.22KiB | 0 | 0.471282 | 17.55MiB | 7.92MiB | 165.38KiB | 0 | 0.451183 | True | True |
syslog_humio_logs | 233.6KiB | 1.42 | 100.00% | 16.02MiB | 394.45KiB | 8.05KiB | 0 | 0.0240427 | 16.25MiB | 377.24KiB | 7.73KiB | 0 | 0.0226707 | False | False |
datadog_agent_remap_datadog_logs | 858.12KiB | 1.4 | 100.00% | 59.88MiB | 2.67MiB | 55.92KiB | 0 | 0.0445522 | 60.72MiB | 3.91MiB | 81.46KiB | 0 | 0.064387 | False | False |
datadog_agent_remap_blackhole | 798.63KiB | 1.4 | 100.00% | 55.85MiB | 5.08MiB | 105.69KiB | 0 | 0.0908737 | 56.63MiB | 4.31MiB | 90.03KiB | 0 | 0.0761678 | False | False |
syslog_splunk_hec_logs | 192.41KiB | 1.17 | 100.00% | 15.99MiB | 750.38KiB | 15.26KiB | 0 | 0.0458046 | 16.18MiB | 718.85KiB | 14.64KiB | 0 | 0.0433704 | False | False |
socket_to_socket_blackhole | 247.94KiB | 1.07 | 100.00% | 22.69MiB | 494.62KiB | 10.1KiB | 0 | 0.0212844 | 22.93MiB | 429.7KiB | 8.77KiB | 0 | 0.0182954 | False | False |
syslog_log2metric_splunk_hec_metrics | 87.7KiB | 0.49 | 99.99% | 17.45MiB | 618.47KiB | 12.61KiB | 0 | 0.0346117 | 17.53MiB | 905.23KiB | 18.41KiB | 0 | 0.050412 | False | False |
splunk_hec_route_s3 | 71.29KiB | 0.38 | 70.30% | 18.44MiB | 2.38MiB | 49.57KiB | 0 | 0.129041 | 18.51MiB | 2.25MiB | 47.07KiB | 0 | 0.121548 | False | False |
syslog_log2metric_humio_metrics | 44.2KiB | 0.34 | 99.98% | 12.83MiB | 231.56KiB | 4.73KiB | 0 | 0.0176223 | 12.87MiB | 540.86KiB | 11.01KiB | 0 | 0.0410227 | False | False |
splunk_hec_indexer_ack_blackhole | 14.73KiB | 0.06 | 44.58% | 23.75MiB | 887.9KiB | 18.06KiB | 0 | 0.0364955 | 23.77MiB | 841.81KiB | 17.14KiB | 0 | 0.0345799 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 5.93KiB | 0.02 | 42.37% | 23.83MiB | 391.6KiB | 7.99KiB | 0 | 0.0160422 | 23.84MiB | 341.88KiB | 6.98KiB | 0 | 0.0140022 | False | False |
enterprise_http_to_http | -1.72KiB | -0.01 | 18.73% | 23.85MiB | 250.15KiB | 5.11KiB | 0 | 0.0102418 | 23.85MiB | 251.08KiB | 5.14KiB | 0 | 0.0102805 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -6.45KiB | -0.03 | 20.93% | 23.76MiB | 831.43KiB | 16.92KiB | 0 | 0.0341709 | 23.75MiB | 859.2KiB | 17.48KiB | 0 | 0.0353216 | False | False |
file_to_blackhole | -101.21KiB | -0.1 | 60.89% | 95.31MiB | 3.64MiB | 75.36KiB | 0 | 0.0381716 | 95.21MiB | 4.37MiB | 90.82KiB | 0 | 0.0458802 | False | False |
http_to_http_noack | -33.59KiB | -0.14 | 87.42% | 23.82MiB | 612.56KiB | 12.52KiB | 0 | 0.0251094 | 23.79MiB | 884.16KiB | 18.02KiB | 0 | 0.0362923 | False | False |
http_to_http_json | -40.8KiB | -0.17 | 99.81% | 23.85MiB | 331.81KiB | 6.77KiB | 0 | 0.0135856 | 23.81MiB | 549.58KiB | 11.22KiB | 0 | 0.0225398 | False | False |
fluent_elasticsearch | -158.32KiB | -0.19 | 100.00% | 79.47MiB | 53.92KiB | 1.09KiB | 0 | 0.000662454 | 79.32MiB | 1.49MiB | 30.63KiB | 0 | 0.0187517 | False | False |
http_pipelines_blackhole_acks | -2.62KiB | -0.21 | 67.67% | 1.23MiB | 109.67KiB | 2.23KiB | 0 | 0.0869383 | 1.23MiB | 70.62KiB | 1.44KiB | 0 | 0.0561004 | False | False |
http_pipelines_blackhole | -5.97KiB | -0.34 | 98.68% | 1.7MiB | 9.22KiB | 193.06B | 0 | 0.00528571 | 1.7MiB | 117.85KiB | 2.4KiB | 0 | 0.0677731 | False | False |
datadog_agent_remap_datadog_logs_acks | -917.7KiB | -1.5 | 100.00% | 59.93MiB | 3.21MiB | 67.13KiB | 0 | 0.0535389 | 59.03MiB | 4.41MiB | 91.77KiB | 0 | 0.0746669 | False | False |
http_pipelines_no_grok_blackhole | -212.22KiB | -1.94 | 100.00% | 10.68MiB | 54.24KiB | 1.11KiB | 0 | 0.00495799 | 10.47MiB | 1.04MiB | 21.65KiB | 0 | 0.0992879 | False | False |
http_text_to_http_json | -1.12MiB | -2.94 | 100.00% | 38.23MiB | 1.24MiB | 25.94KiB | 0 | 0.0324542 | 37.11MiB | 745.5KiB | 15.22KiB | 0 | 0.0196147 | False | False |
Thanks @miton18 for your contribution!