vector icon indicating copy to clipboard operation
vector copied to clipboard

Published 20 hours ago • verified publisher icon vectordotdev

feat(pulsar): message keys as config field

Open miton18 opened this issue 3 years ago • 6 comments
trafficstars

miton18 avatar Sep 21 '22 08:09 miton18

CLA assistant check
All committers have signed the CLA.

bits-bot avatar Sep 21 '22 08:09 bits-bot

Deploy Preview for vector-project canceled.

Name Link
Latest commit ba16a27eaf3c0b03fccc7c8e18f748795e43afe8
Latest deploy log https://app.netlify.com/sites/vector-project/deploys/632c788a05cc5e0009afa400

netlify[bot] avatar Sep 21 '22 08:09 netlify[bot]

Soak Test Results

Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: 2db1721b69d542e4bf1c9a3bf2301c07ce2229aa Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_blackhole_acks 2.84KiB 0.22 67.29% 1.24MiB 111.28KiB 2.26KiB 0 0.0878911 1.24MiB 88.51KiB 1.8KiB 0 0.0697547 False False
splunk_hec_to_splunk_hec_logs_noack 16.7KiB 0.07 83.96% 23.82MiB 476.8KiB 9.73KiB 0 0.0195434 23.84MiB 334.94KiB 6.84KiB 0 0.0137195 False False
enterprise_http_to_http 2.13KiB 0.01 22.50% 23.84MiB 254.75KiB 5.2KiB 0 0.0104318 23.85MiB 260.48KiB 5.33KiB 0 0.0106656 False False
splunk_hec_to_splunk_hec_logs_acks -1.47KiB -0.01 4.47% 23.74MiB 906.88KiB 18.44KiB 0 0.0372987 23.74MiB 915.15KiB 18.61KiB 0 0.0376408 False False
splunk_hec_indexer_ack_blackhole -6.57KiB -0.03 19.98% 23.75MiB 882.58KiB 17.95KiB 0 0.0362834 23.74MiB 923.63KiB 18.78KiB 0 0.0379813 False False
file_to_blackhole -42.25KiB -0.04 31.09% 95.34MiB 3.49MiB 72.26KiB 0 0.036557 95.29MiB 3.7MiB 76.99KiB 0 0.038829 False False
socket_to_socket_blackhole -21.17KiB -0.09 97.00% 23.18MiB 121.41KiB 2.48KiB 0 0.00511429 23.16MiB 461.96KiB 9.43KiB 0 0.0194771 False False
http_to_http_json -40.53KiB -0.17 99.86% 23.85MiB 341.47KiB 6.97KiB 0 0.0139809 23.81MiB 519.63KiB 10.6KiB 0 0.0213111 False False
fluent_elasticsearch -174.95KiB -0.21 100.00% 79.47MiB 53.53KiB 1.08KiB 0 0.000657593 79.3MiB 1.58MiB 32.62KiB 0 0.0199804 False False
syslog_regex_logs2metric_ddmetrics -33.21KiB -0.26 95.89% 12.51MiB 602.98KiB 12.29KiB 0 0.0470521 12.48MiB 522.39KiB 10.65KiB 0 0.0408693 False False
http_to_http_noack -95.22KiB -0.39 100.00% 23.85MiB 253.62KiB 5.19KiB 0 0.0103841 23.75MiB 1.05MiB 21.95KiB 0 0.0443113 False False
datadog_agent_remap_blackhole_acks -343.39KiB -0.57 99.60% 59.04MiB 4.74MiB 98.76KiB 0 0.0803309 58.7MiB 3.2MiB 66.95KiB 0 0.0544216 False False
syslog_splunk_hec_logs -172.73KiB -1.03 100.00% 16.32MiB 752.89KiB 15.32KiB 0 0.0450298 16.16MiB 659.56KiB 13.45KiB 0 0.0398593 False False
datadog_agent_remap_blackhole -650.32KiB -1.05 100.00% 60.29MiB 3.66MiB 76.36KiB 0 0.0607056 59.66MiB 3.16MiB 65.9KiB 0 0.0529442 False False
syslog_log2metric_humio_metrics -139.58KiB -1.1 100.00% 12.34MiB 408.55KiB 8.34KiB 0 0.0323156 12.21MiB 456.19KiB 9.29KiB 0 0.0364866 False False
splunk_hec_route_s3 -234.39KiB -1.21 99.96% 18.98MiB 2.26MiB 47.02KiB 0 0.11898 18.75MiB 2.22MiB 46.5KiB 0 0.118625 False False
datadog_agent_remap_datadog_logs -873.52KiB -1.38 100.00% 61.97MiB 2.54MiB 53.22KiB 0 0.0409348 61.12MiB 4.05MiB 84.31KiB 0 0.0662171 False False
syslog_humio_logs -250.91KiB -1.47 100.00% 16.67MiB 129.33KiB 2.64KiB 0 0.00757718 16.42MiB 121.32KiB 2.48KiB 0 0.00721355 False False
datadog_agent_remap_datadog_logs_acks -984.75KiB -1.54 100.00% 62.32MiB 3.41MiB 71.16KiB 0 0.054662 61.36MiB 4.36MiB 90.68KiB 0 0.0709805 False False
http_to_http_acks -286.55KiB -1.59 78.37% 17.59MiB 8.02MiB 167.61KiB 0 0.455753 17.31MiB 7.66MiB 160.04KiB 0 0.442664 True True
http_pipelines_no_grok_blackhole -181.01KiB -1.62 100.00% 10.88MiB 50.87KiB 1.04KiB 0 0.00456505 10.7MiB 1.06MiB 22.13KiB 0 0.0992963 False False
syslog_log2metric_splunk_hec_metrics -353.33KiB -1.97 100.00% 17.52MiB 689.79KiB 14.06KiB 0 0.0384426 17.17MiB 891.56KiB 18.15KiB 0 0.0506856 False False
http_pipelines_blackhole -41.1KiB -2.33 100.00% 1.72MiB 10.53KiB 220.59B 0 0.00597423 1.68MiB 118.61KiB 2.42KiB 0 0.0688938 False False
syslog_loki -409.0KiB -2.74 100.00% 14.58MiB 422.06KiB 8.63KiB 0 0.0282659 14.18MiB 709.07KiB 14.41KiB 0 0.0488255 False False
http_text_to_http_json -2.06MiB -5.2 100.00% 39.6MiB 748.64KiB 15.28KiB 0 0.0184574 37.54MiB 816.11KiB 16.66KiB 0 0.0212255 False False

github-actions[bot] avatar Sep 21 '22 09:09 github-actions[bot]

Soak Test Results

Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: 1884b926cacde40a73e7087eb93ee18eb51bdfc9 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_to_http_acks 201.51KiB 1.14 61.57% 17.25MiB 8.04MiB 168.22KiB 0 0.466202 17.45MiB 7.63MiB 159.21KiB 0 0.437266 True True
syslog_log2metric_humio_metrics 15.78KiB 0.13 71.02% 12.22MiB 476.44KiB 9.71KiB 0 0.0380729 12.23MiB 555.59KiB 11.31KiB 0 0.0443417 False False
splunk_hec_to_splunk_hec_logs_noack 26.02KiB 0.11 95.65% 23.81MiB 531.31KiB 10.84KiB 0 0.0217858 23.84MiB 341.4KiB 6.97KiB 0 0.0139836 False False
splunk_hec_indexer_ack_blackhole 16.67KiB 0.07 44.89% 23.73MiB 998.41KiB 20.3KiB 0 0.0410822 23.74MiB 945.55KiB 19.23KiB 0 0.0388804 False False
splunk_hec_to_splunk_hec_logs_acks 1.77KiB 0.01 5.71% 23.75MiB 854.37KiB 17.38KiB 0 0.0351215 23.75MiB 859.57KiB 17.48KiB 0 0.0353325 False False
enterprise_http_to_http -778.3B -0 8.23% 23.84MiB 252.2KiB 5.15KiB 0 0.0103267 23.84MiB 257.05KiB 5.26KiB 0 0.0105259 False False
file_to_blackhole -93.41KiB -0.1 65.03% 95.34MiB 3.09MiB 64.14KiB 0 0.0324473 95.25MiB 3.68MiB 76.56KiB 0 0.038671 False False
http_to_http_json -40.2KiB -0.16 99.81% 23.85MiB 329.64KiB 6.73KiB 0 0.0134957 23.81MiB 539.54KiB 11.02KiB 0 0.0221253 False False
fluent_elasticsearch -162.72KiB -0.2 100.00% 79.47MiB 54.94KiB 1.11KiB 0 0.000674912 79.31MiB 1.53MiB 31.46KiB 0 0.0192627 False False
http_pipelines_blackhole_acks -3.34KiB -0.26 79.99% 1.23MiB 107.85KiB 2.2KiB 0 0.0854625 1.23MiB 68.84KiB 1.4KiB 0 0.054694 False False
socket_to_socket_blackhole -69.24KiB -0.3 100.00% 22.92MiB 190.76KiB 3.89KiB 0 0.00812733 22.85MiB 394.1KiB 8.05KiB 0 0.0168404 False False
syslog_regex_logs2metric_ddmetrics -48.13KiB -0.38 99.23% 12.32MiB 633.41KiB 12.91KiB 0 0.0501957 12.27MiB 618.51KiB 12.6KiB 0 0.0492026 False False
http_to_http_noack -101.08KiB -0.41 100.00% 23.84MiB 409.74KiB 8.37KiB 0 0.0167833 23.74MiB 1.12MiB 23.25KiB 0 0.0469744 False False
datadog_agent_remap_blackhole -290.81KiB -0.47 99.23% 60.77MiB 4.05MiB 84.33KiB 0 0.0665791 60.49MiB 3.31MiB 69.08KiB 0 0.0547543 False False
syslog_splunk_hec_logs -157.06KiB -0.94 100.00% 16.34MiB 703.33KiB 14.32KiB 0 0.0420229 16.19MiB 622.17KiB 12.69KiB 0 0.0375261 False False
http_pipelines_blackhole -17.95KiB -1.09 100.00% 1.61MiB 47.24KiB 988.29B 0 0.0285776 1.6MiB 118.46KiB 2.41KiB 0 0.0724472 False False
datadog_agent_remap_blackhole_acks -709.7KiB -1.16 100.00% 59.78MiB 4.84MiB 100.84KiB 0 0.080938 59.09MiB 3.75MiB 78.51KiB 0 0.0634484 False False
syslog_humio_logs -210.19KiB -1.25 100.00% 16.39MiB 476.9KiB 9.74KiB 0 0.0284169 16.18MiB 437.42KiB 8.96KiB 0 0.0263952 False False
syslog_log2metric_splunk_hec_metrics -330.46KiB -1.79 100.00% 18.02MiB 639.55KiB 13.03KiB 0 0.0346604 17.69MiB 691.64KiB 14.09KiB 0 0.0381668 False False
datadog_agent_remap_datadog_logs_acks -1.11MiB -1.82 100.00% 60.96MiB 3.43MiB 71.7KiB 0 0.0562866 59.85MiB 4.44MiB 92.41KiB 0 0.0741664 False False
datadog_agent_remap_datadog_logs -1.19MiB -1.93 100.00% 61.74MiB 1.38MiB 28.86KiB 0 0.022289 60.55MiB 3.96MiB 82.5KiB 0 0.0653809 False False
syslog_loki -533.24KiB -3.62 100.00% 14.4MiB 410.02KiB 8.39KiB 0 0.0277992 13.88MiB 705.16KiB 14.33KiB 0 0.0496028 False False
http_text_to_http_json -2.06MiB -5.21 100.00% 39.52MiB 813.94KiB 16.61KiB 0 0.0201082 37.46MiB 795.21KiB 16.23KiB 0 0.0207247 False False
splunk_hec_route_s3 -1015.68KiB -5.63 100.00% 17.62MiB 2.58MiB 53.82KiB 0 0.146661 16.62MiB 2.64MiB 55.16KiB 0 0.158619 False False
http_pipelines_no_grok_blackhole -664.67KiB -6.27 100.00% 10.35MiB 771.07KiB 15.74KiB 0 0.0727366 9.7MiB 1.36MiB 28.33KiB 0 0.140221 False False

github-actions[bot] avatar Sep 21 '22 09:09 github-actions[bot]

Soak Test Results

Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: 5de4d5556dbf26f46e2966fafd3e4acb134077d3 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_to_http_acks 377.12KiB 2.13 89.14% 17.33MiB 8.11MiB 169.6KiB 0 0.4681 17.7MiB 7.79MiB 162.66KiB 0 0.439878 True True
syslog_log2metric_humio_metrics 31.0KiB 0.24 97.62% 12.61MiB 368.16KiB 7.52KiB 0 0.0284953 12.64MiB 563.46KiB 11.47KiB 0 0.0435072 False False
splunk_hec_indexer_ack_blackhole 39.58KiB 0.16 90.41% 23.74MiB 903.79KiB 18.38KiB 0 0.0371693 23.78MiB 740.38KiB 15.08KiB 0 0.0303994 False False
splunk_hec_to_splunk_hec_logs_noack 7.35KiB 0.03 52.34% 23.83MiB 379.68KiB 7.76KiB 0 0.0155555 23.84MiB 333.94KiB 6.82KiB 0 0.0136773 False False
enterprise_http_to_http -1.04KiB -0 11.43% 23.85MiB 247.59KiB 5.05KiB 0 0.0101374 23.85MiB 251.03KiB 5.13KiB 0 0.0102784 False False
splunk_hec_to_splunk_hec_logs_acks -4.62KiB -0.02 15.48% 23.76MiB 807.11KiB 16.42KiB 0 0.0331646 23.76MiB 835.36KiB 17.0KiB 0 0.0343317 False False
file_to_blackhole -48.04KiB -0.05 32.07% 95.33MiB 3.71MiB 76.87KiB 0 0.03889 95.29MiB 4.19MiB 87.14KiB 0 0.043924 False False
http_to_http_json -37.35KiB -0.15 99.64% 23.85MiB 344.15KiB 7.03KiB 0 0.0140908 23.81MiB 526.29KiB 10.75KiB 0 0.0215811 False False
fluent_elasticsearch -180.57KiB -0.22 100.00% 79.47MiB 54.15KiB 1.1KiB 0 0.00066524 79.3MiB 1.62MiB 33.3KiB 0 0.0204006 False False
datadog_agent_remap_blackhole_acks -236.97KiB -0.37 98.54% 62.37MiB 3.91MiB 81.41KiB 0 0.0626302 62.14MiB 2.52MiB 52.66KiB 0 0.0405131 False False
http_to_http_noack -96.06KiB -0.39 99.97% 23.83MiB 519.44KiB 10.62KiB 0 0.0212836 23.73MiB 1.15MiB 24.0KiB 0 0.0484928 False False
http_pipelines_blackhole_acks -5.66KiB -0.45 95.81% 1.22MiB 108.41KiB 2.21KiB 0 0.0866845 1.22MiB 83.09KiB 1.69KiB 0 0.0667362 False False
syslog_splunk_hec_logs -114.08KiB -0.71 100.00% 15.67MiB 731.55KiB 14.88KiB 0 0.0455836 15.56MiB 605.54KiB 12.36KiB 0 0.038002 False False
syslog_regex_logs2metric_ddmetrics -92.6KiB -0.72 100.00% 12.63MiB 496.11KiB 10.12KiB 0 0.0383534 12.54MiB 569.07KiB 11.59KiB 0 0.0443111 False False
socket_to_socket_blackhole -176.2KiB -0.74 100.00% 23.17MiB 134.32KiB 2.74KiB 0 0.00565905 23.0MiB 482.98KiB 9.86KiB 0 0.0205009 False False
syslog_log2metric_splunk_hec_metrics -175.2KiB -0.99 100.00% 17.36MiB 660.59KiB 13.45KiB 0 0.0371463 17.19MiB 720.83KiB 14.67KiB 0 0.040937 False False
http_pipelines_blackhole -21.68KiB -1.25 100.00% 1.69MiB 12.66KiB 264.76B 0 0.00729118 1.67MiB 126.03KiB 2.57KiB 0 0.0735242 False False
datadog_agent_remap_datadog_logs -873.49KiB -1.37 100.00% 62.32MiB 1.29MiB 26.94KiB 0 0.0206198 61.47MiB 4.15MiB 86.37KiB 0 0.0674579 False False
datadog_agent_remap_blackhole -855.06KiB -1.39 100.00% 60.07MiB 4.16MiB 86.64KiB 0 0.0692442 59.24MiB 3.06MiB 63.92KiB 0 0.0517209 False False
syslog_humio_logs -254.49KiB -1.49 100.00% 16.63MiB 123.63KiB 2.52KiB 0 0.00725923 16.38MiB 126.48KiB 2.59KiB 0 0.0075397 False False
datadog_agent_remap_datadog_logs_acks -946.41KiB -1.51 100.00% 61.27MiB 3.45MiB 71.96KiB 0 0.0562252 60.35MiB 4.32MiB 89.83KiB 0 0.0714977 False False
http_pipelines_no_grok_blackhole -184.77KiB -1.66 100.00% 10.88MiB 57.53KiB 1.17KiB 0 0.00516074 10.7MiB 1.06MiB 22.15KiB 0 0.0993959 False False
splunk_hec_route_s3 -324.63KiB -1.69 100.00% 18.73MiB 2.26MiB 47.03KiB 0 0.120485 18.42MiB 2.22MiB 46.46KiB 0 0.120633 False False
syslog_loki -262.08KiB -1.95 100.00% 13.14MiB 927.81KiB 19.0KiB 0 0.0689576 12.88MiB 1.17MiB 24.44KiB 0 0.0911303 False False
http_text_to_http_json -2.14MiB -5.48 100.00% 39.01MiB 979.02KiB 19.98KiB 0 0.0245061 36.87MiB 830.09KiB 16.95KiB 0 0.021984 False False

github-actions[bot] avatar Sep 21 '22 11:09 github-actions[bot]

Soak Test Results

Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: d146ad5976d323a8a6db7881dde879adbe1ab3ee Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_blackhole_acks 13.67KiB 1.1 100.00% 1.22MiB 109.71KiB 2.23KiB 0 0.0881505 1.23MiB 37.61KiB 785.87B 0 0.0298883 False False
splunk_hec_to_splunk_hec_logs_noack 11.36KiB 0.05 69.05% 23.83MiB 434.95KiB 8.88KiB 0 0.0178224 23.84MiB 332.46KiB 6.79KiB 0 0.0136166 False False
splunk_hec_indexer_ack_blackhole 10.08KiB 0.04 31.75% 23.75MiB 872.86KiB 17.75KiB 0 0.035882 23.76MiB 839.21KiB 17.08KiB 0 0.0344847 False False
enterprise_http_to_http 6.61B 0 0.07% 23.85MiB 249.41KiB 5.09KiB 0 0.0102123 23.85MiB 251.67KiB 5.15KiB 0 0.0103047 False False
file_to_blackhole -61.09KiB -0.06 49.46% 95.35MiB 3.01MiB 62.42KiB 0 0.0315752 95.29MiB 3.23MiB 67.2KiB 0 0.0339227 False False
splunk_hec_to_splunk_hec_logs_acks -15.9KiB -0.07 49.07% 23.76MiB 805.93KiB 16.4KiB 0 0.0331131 23.75MiB 867.87KiB 17.65KiB 0 0.0356814 False False
http_to_http_json -38.9KiB -0.16 99.74% 23.85MiB 329.79KiB 6.73KiB 0 0.0135025 23.81MiB 539.78KiB 11.02KiB 0 0.0221353 False False
fluent_elasticsearch -212.68KiB -0.26 100.00% 79.47MiB 53.18KiB 1.08KiB 0 0.000653314 79.27MiB 1.82MiB 37.55KiB 0 0.023017 False False
http_to_http_noack -95.13KiB -0.39 99.99% 23.84MiB 412.06KiB 8.43KiB 0 0.0168774 23.74MiB 1.09MiB 22.79KiB 0 0.0460266 False False
datadog_agent_remap_blackhole_acks -402.75KiB -0.63 100.00% 62.32MiB 4.26MiB 88.69KiB 0 0.068324 61.93MiB 1.74MiB 36.54KiB 0 0.0281514 False False
splunk_hec_route_s3 -134.54KiB -0.72 95.28% 18.16MiB 2.32MiB 48.35KiB 0 0.127744 18.03MiB 2.27MiB 47.51KiB 0 0.125961 False False
syslog_regex_logs2metric_ddmetrics -95.17KiB -0.75 100.00% 12.39MiB 556.32KiB 11.33KiB 0 0.043836 12.3MiB 524.44KiB 10.69KiB 0 0.0416367 False False
socket_to_socket_blackhole -239.49KiB -1.04 100.00% 22.51MiB 725.46KiB 14.81KiB 0 0.0314596 22.28MiB 947.63KiB 19.35KiB 0 0.0415251 False False
http_to_http_acks -214.76KiB -1.19 64.51% 17.62MiB 8.15MiB 170.36KiB 0 0.462522 17.41MiB 7.55MiB 157.68KiB 0 0.433869 True True
syslog_splunk_hec_logs -208.7KiB -1.23 100.00% 16.62MiB 782.41KiB 15.92KiB 0 0.0459699 16.41MiB 641.47KiB 13.07KiB 0 0.038157 False False
datadog_agent_remap_datadog_logs -971.21KiB -1.54 100.00% 61.4MiB 823.98KiB 16.87KiB 0 0.0131035 60.45MiB 4.07MiB 84.8KiB 0 0.0673503 False False
syslog_log2metric_humio_metrics -200.5KiB -1.58 100.00% 12.43MiB 405.37KiB 8.28KiB 0 0.0318477 12.23MiB 626.32KiB 12.74KiB 0 0.0499943 False False
syslog_humio_logs -265.44KiB -1.59 100.00% 16.29MiB 545.67KiB 11.14KiB 0 0.0327024 16.03MiB 592.5KiB 12.13KiB 0 0.0360834 False False
datadog_agent_remap_datadog_logs_acks -1015.02KiB -1.61 100.00% 61.58MiB 2.94MiB 61.48KiB 0 0.0477358 60.59MiB 4.27MiB 88.9KiB 0 0.0704746 False False
http_pipelines_no_grok_blackhole -182.0KiB -1.64 100.00% 10.81MiB 113.53KiB 2.32KiB 0 0.0102498 10.64MiB 1.03MiB 21.53KiB 0 0.0971789 False False
datadog_agent_remap_blackhole -1.18MiB -2.06 100.00% 57.15MiB 4.05MiB 84.48KiB 0 0.070896 55.97MiB 3.2MiB 66.79KiB 0 0.0571918 False False
http_pipelines_blackhole -40.63KiB -2.31 100.00% 1.72MiB 11.5KiB 240.69B 0 0.00653673 1.68MiB 115.09KiB 2.35KiB 0 0.0669587 False False
syslog_log2metric_splunk_hec_metrics -513.97KiB -2.88 100.00% 17.42MiB 697.7KiB 14.22KiB 0 0.0391041 16.92MiB 796.81KiB 16.22KiB 0 0.0459837 False False
syslog_loki -558.36KiB -3.78 100.00% 14.43MiB 324.42KiB 6.65KiB 0 0.021958 13.88MiB 737.42KiB 14.99KiB 0 0.051873 False False
http_text_to_http_json -2.05MiB -5.2 100.00% 39.44MiB 856.52KiB 17.48KiB 0 0.0212055 37.39MiB 809.11KiB 16.52KiB 0 0.021131 False False

github-actions[bot] avatar Sep 21 '22 11:09 github-actions[bot]

Soak Test Results

Baseline: d8ca608861ee14e572a3a41cc49728678b9b7c91 Comparison: b3d5010970b97286db6c24aae57660276452e438 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_splunk_hec_logs 734.42KiB 4.7 100.00% 15.25MiB 1.14MiB 23.75KiB 0 0.0747066 15.97MiB 1.25MiB 26.05KiB 0 0.0782018 False False
socket_to_socket_blackhole 362.15KiB 1.59 100.00% 22.31MiB 586.12KiB 11.96KiB 0 0.0256501 22.66MiB 451.91KiB 9.23KiB 0 0.019468 False False
syslog_log2metric_humio_metrics 120.88KiB 0.95 100.00% 12.37MiB 281.95KiB 5.76KiB 0 0.0222536 12.49MiB 492.83KiB 10.03KiB 0 0.0385309 False False
syslog_loki 133.56KiB 0.91 100.00% 14.32MiB 472.86KiB 9.68KiB 0 0.0322423 14.45MiB 729.67KiB 14.83KiB 0 0.0493043 False False
http_pipelines_blackhole_acks 8.62KiB 0.71 99.62% 1.18MiB 119.41KiB 2.43KiB 0 0.0986706 1.19MiB 84.23KiB 1.72KiB 0 0.069114 False False
http_to_http_acks 124.14KiB 0.71 40.34% 17.12MiB 7.91MiB 165.31KiB 0 0.461806 17.24MiB 7.98MiB 166.34KiB 0 0.462472 True True
syslog_regex_logs2metric_ddmetrics 84.75KiB 0.66 100.00% 12.46MiB 580.72KiB 11.83KiB 0 0.0455115 12.54MiB 564.09KiB 11.5KiB 0 0.0439164 False False
syslog_humio_logs 71.98KiB 0.43 100.00% 16.41MiB 156.63KiB 3.2KiB 0 0.00931655 16.49MiB 147.89KiB 3.03KiB 0 0.00875876 False False
splunk_hec_to_splunk_hec_logs_noack 21.82KiB 0.09 92.06% 23.82MiB 504.3KiB 10.29KiB 0 0.0206727 23.84MiB 341.96KiB 6.98KiB 0 0.0140056 False False
splunk_hec_route_s3 13.94KiB 0.08 15.76% 18.05MiB 2.39MiB 49.78KiB 0 0.132476 18.06MiB 2.36MiB 49.4KiB 0 0.130788 False False
syslog_log2metric_splunk_hec_metrics 7.47KiB 0.04 25.42% 17.33MiB 791.1KiB 16.13KiB 0 0.0445636 17.34MiB 808.78KiB 16.48KiB 0 0.0455402 False False
splunk_hec_to_splunk_hec_logs_acks 2.26KiB 0.01 7.03% 23.74MiB 892.49KiB 18.15KiB 0 0.0367025 23.74MiB 891.47KiB 18.13KiB 0 0.0366573 False False
enterprise_http_to_http -2.68KiB -0.01 28.53% 23.85MiB 250.18KiB 5.11KiB 0 0.010243 23.84MiB 256.66KiB 5.25KiB 0 0.0105095 False False
splunk_hec_indexer_ack_blackhole -5.83KiB -0.02 18.59% 23.76MiB 851.68KiB 17.33KiB 0 0.0350051 23.75MiB 873.2KiB 17.76KiB 0 0.0358983 False False
file_to_blackhole -100.11KiB -0.1 64.39% 95.34MiB 3.38MiB 69.97KiB 0 0.0353973 95.24MiB 3.99MiB 82.89KiB 0 0.0418876 False False
http_to_http_json -31.68KiB -0.13 98.85% 23.84MiB 343.7KiB 7.02KiB 0 0.0140737 23.81MiB 508.51KiB 10.39KiB 0 0.0208491 False False
fluent_elasticsearch -164.11KiB -0.2 100.00% 79.47MiB 54.37KiB 1.1KiB 0 0.0006679 79.31MiB 1.54MiB 31.69KiB 0 0.0193977 False False
http_to_http_noack -77.3KiB -0.32 99.86% 23.83MiB 522.0KiB 10.66KiB 0 0.0213882 23.75MiB 1.04MiB 21.69KiB 0 0.0437763 False False
datadog_agent_remap_datadog_logs_acks -654.88KiB -1.03 100.00% 61.88MiB 2.9MiB 60.67KiB 0 0.0468576 61.24MiB 4.36MiB 90.77KiB 0 0.0711862 False False
http_pipelines_blackhole -19.39KiB -1.12 100.00% 1.69MiB 14.35KiB 300.52B 0 0.00828462 1.67MiB 115.91KiB 2.36KiB 0 0.0676591 False False
datadog_agent_remap_blackhole_acks -829.67KiB -1.33 100.00% 61.06MiB 3.82MiB 79.65KiB 0 0.0625892 60.25MiB 2.85MiB 59.7KiB 0 0.0473692 False False
datadog_agent_remap_datadog_logs -1007.05KiB -1.58 100.00% 62.42MiB 748.17KiB 15.32KiB 0 0.0117019 61.44MiB 4.3MiB 89.5KiB 0 0.0699537 False False
datadog_agent_remap_blackhole -1.03MiB -1.76 100.00% 58.68MiB 4.3MiB 89.65KiB 0 0.0733091 57.65MiB 3.76MiB 78.52KiB 0 0.0652841 False False
http_pipelines_no_grok_blackhole -229.66KiB -2.11 100.00% 10.65MiB 77.58KiB 1.58KiB 0 0.00711313 10.42MiB 1.01MiB 21.01KiB 0 0.0967673 False False
http_text_to_http_json -1.49MiB -3.83 100.00% 38.96MiB 1.22MiB 25.54KiB 0 0.0313566 37.47MiB 814.04KiB 16.62KiB 0 0.0212112 False False

github-actions[bot] avatar Sep 22 '22 14:09 github-actions[bot]

I'm discovering cue scripts, thanks @fuchsnj

miton18 avatar Sep 22 '22 14:09 miton18

Soak Test Results

Baseline: 248583fe27dfb3c1393c80580af5c3340a89cd3f Comparison: cd5b3112cacad16c2632efd2dc38608865dc9c41 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
socket_to_socket_blackhole 640.32KiB 2.89 100.00% 21.66MiB 747.23KiB 15.25KiB 0 0.03369 22.28MiB 672.42KiB 13.73KiB 0 0.0294661 False False
syslog_log2metric_humio_metrics 248.42KiB 1.98 100.00% 12.25MiB 579.48KiB 11.83KiB 0 0.0461997 12.49MiB 603.89KiB 12.3KiB 0 0.0472109 False False
syslog_splunk_hec_logs 214.23KiB 1.3 100.00% 16.11MiB 654.6KiB 13.34KiB 0 0.0396714 16.32MiB 588.67KiB 12.0KiB 0 0.0352187 False False
syslog_humio_logs 104.61KiB 0.62 100.00% 16.36MiB 256.85KiB 5.24KiB 0 0.0153296 16.46MiB 192.7KiB 3.95KiB 0 0.0114299 False False
syslog_regex_logs2metric_ddmetrics 73.66KiB 0.58 100.00% 12.49MiB 587.63KiB 11.97KiB 0 0.0459319 12.56MiB 516.11KiB 10.52KiB 0 0.0401104 False False
syslog_loki 70.58KiB 0.48 100.00% 14.43MiB 441.16KiB 9.03KiB 0 0.0298403 14.5MiB 718.38KiB 14.6KiB 0 0.0483613 False False
http_pipelines_blackhole_acks 4.45KiB 0.36 90.64% 1.22MiB 99.56KiB 2.03KiB 0 0.079622 1.23MiB 84.0KiB 1.71KiB 0 0.06694 False False
splunk_hec_to_splunk_hec_logs_noack 12.42KiB 0.05 73.33% 23.83MiB 432.72KiB 8.84KiB 0 0.0177324 23.84MiB 335.37KiB 6.85KiB 0 0.0137364 False False
enterprise_http_to_http -1.61KiB -0.01 17.43% 23.85MiB 248.82KiB 5.08KiB 0 0.0101872 23.85MiB 257.23KiB 5.26KiB 0 0.0105321 False False
splunk_hec_to_splunk_hec_logs_acks -5.0KiB -0.02 15.66% 23.75MiB 870.39KiB 17.7KiB 0 0.0357832 23.74MiB 887.97KiB 18.06KiB 0 0.0365136 False False
file_to_blackhole -50.94KiB -0.05 46.28% 95.35MiB 2.73MiB 56.63KiB 0 0.0286449 95.3MiB 2.89MiB 60.08KiB 0 0.0302927 False False
splunk_hec_indexer_ack_blackhole -23.95KiB -0.1 63.99% 23.76MiB 858.86KiB 17.48KiB 0 0.0352868 23.74MiB 957.68KiB 19.48KiB 0 0.0393856 False False
http_to_http_json -26.65KiB -0.11 97.61% 23.85MiB 334.18KiB 6.82KiB 0 0.0136822 23.82MiB 470.45KiB 9.62KiB 0 0.0192825 False False
syslog_log2metric_splunk_hec_metrics -19.49KiB -0.11 52.67% 17.19MiB 910.11KiB 18.56KiB 0 0.0516841 17.17MiB 975.18KiB 19.86KiB 0 0.0554409 False False
http_to_http_noack -68.54KiB -0.28 99.96% 23.85MiB 261.66KiB 5.35KiB 0 0.0107133 23.78MiB 917.96KiB 18.71KiB 0 0.0376901 False False
fluent_elasticsearch -223.93KiB -0.28 100.00% 79.47MiB 52.7KiB 1.06KiB 0 0.000647437 79.25MiB 2.5MiB 51.39KiB 0 0.0315038 False False
datadog_agent_remap_blackhole_acks -305.66KiB -0.51 96.47% 57.97MiB 5.3MiB 110.41KiB 0 0.091437 57.67MiB 4.52MiB 94.32KiB 0 0.0782681 False False
datadog_agent_remap_datadog_logs -404.33KiB -0.66 100.00% 60.23MiB 1.81MiB 37.91KiB 0 0.0300135 59.84MiB 4.19MiB 87.26KiB 0 0.069981 False False
datadog_agent_remap_datadog_logs_acks -500.19KiB -0.8 100.00% 61.16MiB 3.95MiB 82.35KiB 0 0.0645637 60.67MiB 4.37MiB 90.96KiB 0 0.0720085 False False
http_to_http_acks -166.06KiB -0.94 51.43% 17.24MiB 8.04MiB 168.12KiB 0 0.466418 17.08MiB 8.09MiB 168.74KiB 0 0.473569 True True
http_pipelines_blackhole -19.7KiB -1.14 100.00% 1.69MiB 16.35KiB 342.34B 0 0.00944109 1.67MiB 119.26KiB 2.43KiB 0 0.0696432 False False
splunk_hec_route_s3 -200.51KiB -1.15 99.15% 17.06MiB 2.6MiB 54.16KiB 0 0.152384 16.86MiB 2.56MiB 53.62KiB 0 0.152018 False False
http_pipelines_no_grok_blackhole -140.19KiB -1.33 100.00% 10.32MiB 292.93KiB 5.98KiB 0 0.0277046 10.19MiB 1014.36KiB 20.63KiB 0 0.0972266 False False
datadog_agent_remap_blackhole -1.13MiB -1.93 100.00% 58.86MiB 3.88MiB 80.93KiB 0 0.0659498 57.72MiB 3.16MiB 65.96KiB 0 0.05477 False False
http_text_to_http_json -1.38MiB -3.56 100.00% 38.72MiB 1.24MiB 26.01KiB 0 0.0321284 37.34MiB 788.4KiB 16.1KiB 0 0.0206143 False False

github-actions[bot] avatar Sep 22 '22 16:09 github-actions[bot]

Soak Test Results

Baseline: 966cd18412bcb0d86a186734690a255758421f0a Comparison: ba16a27eaf3c0b03fccc7c8e18f748795e43afe8 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_regex_logs2metric_ddmetrics 493.84KiB 4.27 100.00% 11.3MiB 788.78KiB 16.07KiB 0 0.0681234 11.79MiB 703.62KiB 14.34KiB 0 0.0582822 False False
syslog_loki 459.46KiB 3.19 100.00% 14.07MiB 423.09KiB 8.67KiB 0 0.0293617 14.52MiB 723.02KiB 14.7KiB 0 0.0486262 False False
datadog_agent_remap_blackhole_acks 1.77MiB 2.99 100.00% 59.02MiB 5.84MiB 121.53KiB 0 0.0988985 60.79MiB 3.58MiB 74.72KiB 0 0.0588189 False False
http_to_http_acks 392.61KiB 2.23 90.29% 17.17MiB 8.09MiB 169.22KiB 0 0.471282 17.55MiB 7.92MiB 165.38KiB 0 0.451183 True True
syslog_humio_logs 233.6KiB 1.42 100.00% 16.02MiB 394.45KiB 8.05KiB 0 0.0240427 16.25MiB 377.24KiB 7.73KiB 0 0.0226707 False False
datadog_agent_remap_datadog_logs 858.12KiB 1.4 100.00% 59.88MiB 2.67MiB 55.92KiB 0 0.0445522 60.72MiB 3.91MiB 81.46KiB 0 0.064387 False False
datadog_agent_remap_blackhole 798.63KiB 1.4 100.00% 55.85MiB 5.08MiB 105.69KiB 0 0.0908737 56.63MiB 4.31MiB 90.03KiB 0 0.0761678 False False
syslog_splunk_hec_logs 192.41KiB 1.17 100.00% 15.99MiB 750.38KiB 15.26KiB 0 0.0458046 16.18MiB 718.85KiB 14.64KiB 0 0.0433704 False False
socket_to_socket_blackhole 247.94KiB 1.07 100.00% 22.69MiB 494.62KiB 10.1KiB 0 0.0212844 22.93MiB 429.7KiB 8.77KiB 0 0.0182954 False False
syslog_log2metric_splunk_hec_metrics 87.7KiB 0.49 99.99% 17.45MiB 618.47KiB 12.61KiB 0 0.0346117 17.53MiB 905.23KiB 18.41KiB 0 0.050412 False False
splunk_hec_route_s3 71.29KiB 0.38 70.30% 18.44MiB 2.38MiB 49.57KiB 0 0.129041 18.51MiB 2.25MiB 47.07KiB 0 0.121548 False False
syslog_log2metric_humio_metrics 44.2KiB 0.34 99.98% 12.83MiB 231.56KiB 4.73KiB 0 0.0176223 12.87MiB 540.86KiB 11.01KiB 0 0.0410227 False False
splunk_hec_indexer_ack_blackhole 14.73KiB 0.06 44.58% 23.75MiB 887.9KiB 18.06KiB 0 0.0364955 23.77MiB 841.81KiB 17.14KiB 0 0.0345799 False False
splunk_hec_to_splunk_hec_logs_noack 5.93KiB 0.02 42.37% 23.83MiB 391.6KiB 7.99KiB 0 0.0160422 23.84MiB 341.88KiB 6.98KiB 0 0.0140022 False False
enterprise_http_to_http -1.72KiB -0.01 18.73% 23.85MiB 250.15KiB 5.11KiB 0 0.0102418 23.85MiB 251.08KiB 5.14KiB 0 0.0102805 False False
splunk_hec_to_splunk_hec_logs_acks -6.45KiB -0.03 20.93% 23.76MiB 831.43KiB 16.92KiB 0 0.0341709 23.75MiB 859.2KiB 17.48KiB 0 0.0353216 False False
file_to_blackhole -101.21KiB -0.1 60.89% 95.31MiB 3.64MiB 75.36KiB 0 0.0381716 95.21MiB 4.37MiB 90.82KiB 0 0.0458802 False False
http_to_http_noack -33.59KiB -0.14 87.42% 23.82MiB 612.56KiB 12.52KiB 0 0.0251094 23.79MiB 884.16KiB 18.02KiB 0 0.0362923 False False
http_to_http_json -40.8KiB -0.17 99.81% 23.85MiB 331.81KiB 6.77KiB 0 0.0135856 23.81MiB 549.58KiB 11.22KiB 0 0.0225398 False False
fluent_elasticsearch -158.32KiB -0.19 100.00% 79.47MiB 53.92KiB 1.09KiB 0 0.000662454 79.32MiB 1.49MiB 30.63KiB 0 0.0187517 False False
http_pipelines_blackhole_acks -2.62KiB -0.21 67.67% 1.23MiB 109.67KiB 2.23KiB 0 0.0869383 1.23MiB 70.62KiB 1.44KiB 0 0.0561004 False False
http_pipelines_blackhole -5.97KiB -0.34 98.68% 1.7MiB 9.22KiB 193.06B 0 0.00528571 1.7MiB 117.85KiB 2.4KiB 0 0.0677731 False False
datadog_agent_remap_datadog_logs_acks -917.7KiB -1.5 100.00% 59.93MiB 3.21MiB 67.13KiB 0 0.0535389 59.03MiB 4.41MiB 91.77KiB 0 0.0746669 False False
http_pipelines_no_grok_blackhole -212.22KiB -1.94 100.00% 10.68MiB 54.24KiB 1.11KiB 0 0.00495799 10.47MiB 1.04MiB 21.65KiB 0 0.0992879 False False
http_text_to_http_json -1.12MiB -2.94 100.00% 38.23MiB 1.24MiB 25.94KiB 0 0.0324542 37.11MiB 745.5KiB 15.22KiB 0 0.0196147 False False

github-actions[bot] avatar Sep 22 '22 16:09 github-actions[bot]

Thanks @miton18 for your contribution!

fuchsnj avatar Sep 23 '22 16:09 fuchsnj