vector
vector copied to clipboard
vectordotdev
enhancement(apt platform,debian platform,dpkg platform,centos platform,rhel platform,rpm platform,yum platform): Improve packaging and service best practices
Closes: https://github.com/vectordotdev/vector/issues/12552 Closes: https://github.com/vectordotdev/vector/issues/12551
The original intent of this work was to adhere to the target spec, though after discussion with other teams at Datadog we were leaning too far into secure by default and harming the UX and onboarding process.
The work here represents a general cleanup of the SystemD service, using more modern and (as far as I can tell) best practices. This pushed some changes to the linux packages (removing now unused files for example).
This also changes the default configuration for Vector to be an Aggregator (the old default did nothing but generate dummy logs into the system's journald ⛔ ), and provides an alternative Agent configuration available via systemd override configurations.
EDIT: Aug 7
Vector should no longer ship a default config, it doesn’t make sense given the tool it’s too flexible to provide a useful default, prefer to add complete and useful example configs - stop including “/etc/vector/vector.toml” in distributed packages.
- SystemD, no default - without args/opts Vector errors
Deploy Preview for vector-project ready!
| Name | Link |
|---|---|
| Latest commit | 971ab7c608f1a5ad9783f1b3f5335f7eb97b5885 |
| Latest deploy log | https://app.netlify.com/sites/vector-project/deploys/62fb99cba441860008fdd84d |
| Deploy Preview | https://deploy-preview-13456--vector-project.netlify.app |
| Preview on mobile | Toggle QR Code...Use your smartphone camera to open QR code link. |
To edit notification comments on pull requests, go to your Netlify site settings.
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
Soak Test Results
Baseline: 5232a05032b709c6951c901c1fcab4f6abedc4b7 Comparison: 5f6d42a99be02c418a47b2766c67df592d58bcd3 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_to_http_acks | 397.9KiB | 2.22 | 90.51% | 17.49MiB | 8.16MiB | 170.54KiB | 0 | 0.46631 | 17.88MiB | 7.96MiB | 166.32KiB | 0 | 0.445172 | True | True |
| splunk_hec_route_s3 | 197.48KiB | 1 | 99.79% | 19.34MiB | 2.24MiB | 46.68KiB | 0 | 0.115959 | 19.53MiB | 2.1MiB | 44.01KiB | 0 | 0.107672 | False | False |
| http_pipelines_blackhole_acks | 9.13KiB | 0.76 | 99.88% | 1.17MiB | 117.87KiB | 2.4KiB | 0 | 0.09839 | 1.18MiB | 72.31KiB | 1.47KiB | 0 | 0.0599034 | False | False |
| socket_to_socket_blackhole | 60.68KiB | 0.45 | 100.00% | 13.27MiB | 447.15KiB | 9.13KiB | 0 | 0.0329003 | 13.33MiB | 478.45KiB | 9.77KiB | 0 | 0.0350473 | False | False |
| syslog_regex_logs2metric_ddmetrics | 50.02KiB | 0.41 | 99.89% | 11.99MiB | 533.12KiB | 10.86KiB | 0 | 0.0433987 | 12.04MiB | 527.52KiB | 10.75KiB | 0 | 0.0427693 | False | False |
| syslog_splunk_hec_logs | 39.55KiB | 0.24 | 90.01% | 16.01MiB | 881.83KiB | 17.96KiB | 0 | 0.0537694 | 16.05MiB | 783.39KiB | 15.98KiB | 0 | 0.0476519 | False | False |
| http_text_to_http_json | 42.79KiB | 0.11 | 78.96% | 39.05MiB | 1.15MiB | 23.95KiB | 0 | 0.0293317 | 39.09MiB | 1.17MiB | 24.36KiB | 0 | 0.02981 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 21.6KiB | 0.09 | 92.15% | 23.82MiB | 503.9KiB | 10.28KiB | 0 | 0.0206579 | 23.84MiB | 327.86KiB | 6.69KiB | 0 | 0.0134292 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 16.75KiB | 0.07 | 50.70% | 23.74MiB | 886.17KiB | 18.02KiB | 0 | 0.0364395 | 23.76MiB | 811.26KiB | 16.51KiB | 0 | 0.0333361 | False | False |
| syslog_humio_logs | 11.81KiB | 0.07 | 62.36% | 15.85MiB | 450.69KiB | 9.21KiB | 0 | 0.027755 | 15.87MiB | 472.7KiB | 9.67KiB | 0 | 0.029089 | False | False |
| splunk_hec_indexer_ack_blackhole | 8.84KiB | 0.04 | 27.11% | 23.75MiB | 900.47KiB | 18.31KiB | 0 | 0.0370243 | 23.75MiB | 872.36KiB | 17.75KiB | 0 | 0.0358554 | False | False |
| file_to_blackhole | -71.43KiB | -0.07 | 44.21% | 95.33MiB | 3.81MiB | 78.93KiB | 0 | 0.03993 | 95.26MiB | 4.46MiB | 92.9KiB | 0 | 0.0468519 | False | False |
| http_to_http_json | -29.26KiB | -0.12 | 98.12% | 23.84MiB | 343.44KiB | 7.01KiB | 0 | 0.0140627 | 23.82MiB | 503.08KiB | 10.28KiB | 0 | 0.0206241 | False | False |
| syslog_loki | -24.03KiB | -0.16 | 86.93% | 14.56MiB | 293.29KiB | 6.01KiB | 0 | 0.0196677 | 14.54MiB | 724.01KiB | 14.72KiB | 0 | 0.0486299 | False | False |
| fluent_elasticsearch | -235.05KiB | -0.29 | 100.00% | 79.47MiB | 53.09KiB | 1.07KiB | 0 | 0.000652251 | 79.24MiB | 2.31MiB | 47.56KiB | 0 | 0.0291585 | False | False |
| http_to_http_noack | -85.29KiB | -0.35 | 99.97% | 23.84MiB | 410.12KiB | 8.38KiB | 0 | 0.0167997 | 23.75MiB | 1.06MiB | 22.01KiB | 0 | 0.0444182 | False | False |
| datadog_agent_remap_blackhole | -311.92KiB | -0.46 | 99.41% | 65.63MiB | 4.69MiB | 97.6KiB | 0 | 0.0714138 | 65.32MiB | 2.75MiB | 57.32KiB | 0 | 0.0420239 | False | False |
| syslog_log2metric_humio_metrics | -63.77KiB | -0.48 | 100.00% | 12.96MiB | 332.01KiB | 6.77KiB | 0 | 0.0250178 | 12.89MiB | 528.21KiB | 10.76KiB | 0 | 0.0399947 | False | False |
| datadog_agent_remap_datadog_logs_acks | -447.44KiB | -0.66 | 99.99% | 65.96MiB | 2.98MiB | 62.25KiB | 0 | 0.0450994 | 65.53MiB | 4.65MiB | 96.71KiB | 0 | 0.0708908 | False | False |
| http_pipelines_blackhole | -10.79KiB | -0.67 | 99.98% | 1.57MiB | 67.22KiB | 1.37KiB | 0 | 0.0417961 | 1.56MiB | 126.91KiB | 2.59KiB | 0 | 0.0794453 | False | False |
| datadog_agent_remap_datadog_logs | -518.6KiB | -0.77 | 100.00% | 65.96MiB | 1.12MiB | 23.48KiB | 0 | 0.0169841 | 65.46MiB | 4.63MiB | 96.39KiB | 0 | 0.0707261 | False | False |
| syslog_log2metric_splunk_hec_metrics | -174.74KiB | -0.99 | 100.00% | 17.17MiB | 889.73KiB | 18.13KiB | 0 | 0.050601 | 17.0MiB | 1.02MiB | 21.3KiB | 0 | 0.0601045 | False | False |
| http_pipelines_no_grok_blackhole | -226.29KiB | -1.93 | 100.00% | 11.47MiB | 499.84KiB | 10.2KiB | 0 | 0.0425436 | 11.25MiB | 1.18MiB | 24.48KiB | 0 | 0.104513 | False | False |
| datadog_agent_remap_blackhole_acks | -1.28MiB | -2.06 | 100.00% | 62.3MiB | 7.08MiB | 147.48KiB | 0 | 0.113652 | 61.02MiB | 6.53MiB | 136.6KiB | 0 | 0.107033 | False | False |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Soak Test Results
Baseline: 2d62397824c9a0ccaf6d678e66b63a4f44bfe4e4 Comparison: 86be36a3bba59b635bc0d310de6cfb6649b6cdc1 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_splunk_hec_logs | 328.3KiB | 1.97 | 100.00% | 16.28MiB | 727.06KiB | 14.8KiB | 0 | 0.0435924 | 16.6MiB | 669.95KiB | 13.65KiB | 0 | 0.0393925 | False | False |
| syslog_regex_logs2metric_ddmetrics | 145.2KiB | 1.15 | 100.00% | 12.33MiB | 590.17KiB | 12.02KiB | 0 | 0.0467356 | 12.47MiB | 522.05KiB | 10.64KiB | 0 | 0.0408707 | False | False |
| syslog_humio_logs | 163.61KiB | 1.02 | 100.00% | 15.68MiB | 231.45KiB | 4.72KiB | 0 | 0.0144084 | 15.84MiB | 275.31KiB | 5.64KiB | 0 | 0.016966 | False | False |
| http_to_http_acks | 147.55KiB | 0.8 | 48.66% | 17.92MiB | 7.65MiB | 159.97KiB | 0 | 0.426797 | 18.07MiB | 7.62MiB | 159.26KiB | 0 | 0.421769 | True | True |
| syslog_log2metric_splunk_hec_metrics | 128.12KiB | 0.77 | 100.00% | 16.35MiB | 581.72KiB | 11.85KiB | 0 | 0.0347366 | 16.48MiB | 727.11KiB | 14.8KiB | 0 | 0.0430887 | False | False |
| syslog_loki | 98.17KiB | 0.67 | 100.00% | 14.33MiB | 638.63KiB | 13.05KiB | 0 | 0.0435225 | 14.42MiB | 841.62KiB | 17.11KiB | 0 | 0.0569756 | False | False |
| socket_to_socket_blackhole | 92.02KiB | 0.66 | 100.00% | 13.58MiB | 240.15KiB | 4.9KiB | 0 | 0.0172618 | 13.67MiB | 167.68KiB | 3.42KiB | 0 | 0.0119738 | False | False |
| splunk_hec_route_s3 | 123.46KiB | 0.63 | 94.48% | 19.16MiB | 2.2MiB | 45.82KiB | 0 | 0.114748 | 19.28MiB | 2.16MiB | 45.22KiB | 0 | 0.112036 | False | False |
| http_pipelines_blackhole_acks | 6.36KiB | 0.53 | 98.68% | 1.17MiB | 106.55KiB | 2.17KiB | 0 | 0.0887905 | 1.18MiB | 67.34KiB | 1.37KiB | 0 | 0.0558207 | False | False |
| syslog_log2metric_humio_metrics | 30.2KiB | 0.22 | 98.33% | 13.31MiB | 267.06KiB | 5.45KiB | 0 | 0.0195889 | 13.34MiB | 558.75KiB | 11.37KiB | 0 | 0.0408939 | False | False |
| datadog_agent_remap_blackhole | 82.79KiB | 0.12 | 50.12% | 65.95MiB | 4.27MiB | 88.96KiB | 0 | 0.0647111 | 66.03MiB | 4.03MiB | 84.05KiB | 0 | 0.0610616 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 25.76KiB | 0.11 | 71.47% | 23.74MiB | 887.8KiB | 18.05KiB | 0 | 0.0365129 | 23.77MiB | 784.73KiB | 15.97KiB | 0 | 0.0322396 | False | False |
| datadog_agent_remap_blackhole_acks | 70.15KiB | 0.1 | 47.97% | 66.61MiB | 4.54MiB | 94.44KiB | 0 | 0.0680722 | 66.68MiB | 2.61MiB | 54.61KiB | 0 | 0.0391615 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 14.77KiB | 0.06 | 80.12% | 23.82MiB | 456.56KiB | 9.32KiB | 0 | 0.0187129 | 23.84MiB | 329.6KiB | 6.73KiB | 0 | 0.013501 | False | False |
| splunk_hec_indexer_ack_blackhole | -2.2KiB | -0.01 | 6.79% | 23.75MiB | 892.65KiB | 18.16KiB | 0 | 0.0366973 | 23.75MiB | 904.19KiB | 18.39KiB | 0 | 0.0371751 | False | False |
| file_to_blackhole | -33.58KiB | -0.03 | 37.34% | 95.36MiB | 2.28MiB | 47.21KiB | 0 | 0.0238741 | 95.33MiB | 2.42MiB | 50.34KiB | 0 | 0.025377 | False | False |
| http_to_http_json | -31.09KiB | -0.13 | 98.88% | 23.85MiB | 332.68KiB | 6.79KiB | 0 | 0.0136207 | 23.82MiB | 499.02KiB | 10.2KiB | 0 | 0.0204572 | False | False |
| http_to_http_noack | -93.89KiB | -0.38 | 100.00% | 23.85MiB | 250.03KiB | 5.11KiB | 0 | 0.0102377 | 23.75MiB | 1.05MiB | 21.83KiB | 0 | 0.0440516 | False | False |
| fluent_elasticsearch | -542.17KiB | -0.67 | 100.00% | 79.47MiB | 52.87KiB | 1.07KiB | 0 | 0.000649491 | 78.94MiB | 5.2MiB | 106.67KiB | 0 | 0.0658574 | False | False |
| http_pipelines_blackhole | -12.46KiB | -0.75 | 100.00% | 1.63MiB | 11.01KiB | 230.5B | 0 | 0.00659238 | 1.62MiB | 97.43KiB | 1.99KiB | 0 | 0.0587926 | False | False |
| http_pipelines_no_grok_blackhole | -134.29KiB | -1.14 | 100.00% | 11.52MiB | 130.93KiB | 2.67KiB | 0 | 0.0111006 | 11.39MiB | 966.78KiB | 19.69KiB | 0 | 0.0829092 | False | False |
| datadog_agent_remap_datadog_logs_acks | -1.06MiB | -1.59 | 100.00% | 66.91MiB | 3.04MiB | 63.5KiB | 0 | 0.0453549 | 65.85MiB | 4.69MiB | 97.58KiB | 0 | 0.0711774 | False | False |
| http_text_to_http_json | -719.97KiB | -1.72 | 100.00% | 40.8MiB | 674.77KiB | 13.77KiB | 0 | 0.0161464 | 40.1MiB | 686.43KiB | 14.03KiB | 0 | 0.0167135 | False | False |
| datadog_agent_remap_datadog_logs | -1.25MiB | -1.93 | 100.00% | 64.89MiB | 1.94MiB | 40.67KiB | 0 | 0.0298776 | 63.63MiB | 4.42MiB | 91.99KiB | 0 | 0.069379 | False | False |
Soak Test Results
Baseline: d52101dea28a7da8a9a5c5fd9159540eba851739 Comparison: 2f8d62044cd7ff702ddd80a05b92f9a9472348af Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole | 1.22MiB | 1.9 | 100.00% | 64.44MiB | 5.55MiB | 115.73KiB | 0 | 0.0861316 | 65.66MiB | 4.41MiB | 91.98KiB | 0 | 0.067114 | False | False |
| http_pipelines_blackhole_acks | 15.03KiB | 1.32 | 100.00% | 1.11MiB | 106.42KiB | 2.17KiB | 0 | 0.0932588 | 1.13MiB | 74.62KiB | 1.52KiB | 0 | 0.0645382 | False | False |
| http_to_http_acks | 183.28KiB | 1 | 59.71% | 17.83MiB | 7.52MiB | 157.15KiB | 0 | 0.42145 | 18.01MiB | 7.32MiB | 152.7KiB | 0 | 0.406371 | True | True |
| http_text_to_http_json | 275.14KiB | 0.67 | 100.00% | 40.07MiB | 695.58KiB | 14.2KiB | 0 | 0.0169493 | 40.34MiB | 717.81KiB | 14.65KiB | 0 | 0.0173745 | False | False |
| datadog_agent_remap_blackhole_acks | 292.57KiB | 0.42 | 98.55% | 67.92MiB | 4.63MiB | 96.41KiB | 0 | 0.068156 | 68.2MiB | 3.39MiB | 70.87KiB | 0 | 0.0496512 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 26.05KiB | 0.11 | 95.76% | 23.81MiB | 538.82KiB | 10.99KiB | 0 | 0.0220934 | 23.84MiB | 324.43KiB | 6.62KiB | 0 | 0.0132887 | False | False |
| datadog_agent_remap_datadog_logs | 51.67KiB | 0.08 | 41.32% | 65.29MiB | 542.44KiB | 11.11KiB | 0 | 0.00811227 | 65.34MiB | 4.54MiB | 94.42KiB | 0 | 0.0693974 | False | False |
| splunk_hec_indexer_ack_blackhole | 7.2KiB | 0.03 | 20.91% | 23.74MiB | 959.89KiB | 19.52KiB | 0 | 0.0394785 | 23.75MiB | 927.32KiB | 18.86KiB | 0 | 0.0381278 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -9.19KiB | -0.04 | 28.36% | 23.75MiB | 858.69KiB | 17.47KiB | 0 | 0.0353029 | 23.74MiB | 899.64KiB | 18.29KiB | 0 | 0.0370005 | False | False |
| file_to_blackhole | -47.47KiB | -0.05 | 32.21% | 95.33MiB | 3.71MiB | 76.99KiB | 0 | 0.0389496 | 95.29MiB | 4.06MiB | 84.44KiB | 0 | 0.04259 | False | False |
| http_to_http_json | -40.97KiB | -0.17 | 99.82% | 23.85MiB | 334.58KiB | 6.83KiB | 0 | 0.0136991 | 23.81MiB | 549.47KiB | 11.22KiB | 0 | 0.0225353 | False | False |
| fluent_elasticsearch | -142.54KiB | -0.18 | 99.98% | 79.47MiB | 53.42KiB | 1.08KiB | 0 | 0.000656235 | 79.33MiB | 1.88MiB | 38.76KiB | 0 | 0.0237187 | False | False |
| datadog_agent_remap_datadog_logs_acks | -266.46KiB | -0.39 | 97.59% | 66.27MiB | 3.17MiB | 66.35KiB | 0 | 0.0478707 | 66.01MiB | 4.69MiB | 97.71KiB | 0 | 0.0710953 | False | False |
| http_pipelines_blackhole | -6.94KiB | -0.41 | 99.97% | 1.64MiB | 13.85KiB | 289.88B | 0 | 0.00822349 | 1.64MiB | 93.61KiB | 1.91KiB | 0 | 0.0558198 | False | False |
| http_to_http_noack | -112.68KiB | -0.46 | 100.00% | 23.84MiB | 402.81KiB | 8.24KiB | 0 | 0.0164996 | 23.73MiB | 1.19MiB | 24.76KiB | 0 | 0.0500685 | False | False |
| socket_to_socket_blackhole | -68.88KiB | -0.49 | 100.00% | 13.6MiB | 211.08KiB | 4.31KiB | 0 | 0.0151574 | 13.53MiB | 184.9KiB | 3.77KiB | 0 | 0.0133437 | False | False |
| syslog_regex_logs2metric_ddmetrics | -71.9KiB | -0.58 | 100.00% | 12.18MiB | 610.09KiB | 12.43KiB | 0 | 0.0489168 | 12.11MiB | 573.42KiB | 11.68KiB | 0 | 0.0462433 | False | False |
| syslog_humio_logs | -151.96KiB | -0.93 | 100.00% | 15.94MiB | 237.57KiB | 4.85KiB | 0 | 0.0145564 | 15.79MiB | 222.14KiB | 4.55KiB | 0 | 0.0137385 | False | False |
| syslog_splunk_hec_logs | -155.31KiB | -0.93 | 100.00% | 16.38MiB | 770.05KiB | 15.67KiB | 0 | 0.0459089 | 16.23MiB | 693.19KiB | 14.12KiB | 0 | 0.0417131 | False | False |
| http_pipelines_no_grok_blackhole | -117.14KiB | -0.99 | 100.00% | 11.52MiB | 381.93KiB | 7.8KiB | 0 | 0.0323598 | 11.41MiB | 1.1MiB | 22.95KiB | 0 | 0.0965735 | False | False |
| syslog_log2metric_humio_metrics | -186.06KiB | -1.36 | 100.00% | 13.4MiB | 200.65KiB | 4.1KiB | 0 | 0.0146183 | 13.22MiB | 540.18KiB | 11.0KiB | 0 | 0.0398961 | False | False |
| splunk_hec_route_s3 | -458.86KiB | -2.3 | 100.00% | 19.44MiB | 2.21MiB | 45.98KiB | 0 | 0.113506 | 19.0MiB | 2.27MiB | 47.44KiB | 0 | 0.119505 | False | False |
| syslog_log2metric_splunk_hec_metrics | -505.87KiB | -2.77 | 100.00% | 17.86MiB | 529.12KiB | 10.78KiB | 0 | 0.028918 | 17.37MiB | 946.89KiB | 19.28KiB | 0 | 0.0532222 | False | False |
| syslog_loki | -652.51KiB | -4.15 | 100.00% | 15.34MiB | 450.78KiB | 9.22KiB | 0 | 0.028695 | 14.7MiB | 847.58KiB | 17.23KiB | 0 | 0.0562932 | False | False |
Also remembered https://github.com/vectordotdev/vector/issues/13209 needs to be addressed - and it looks like we copy it to the same place for debs?
Also remembered #13209 needs to be addressed - and it looks like we copy it to the same place for debs?
I think it's ok to leave that issue aside for now since it out-of-scope of the target spec. We can follow up on it.
Also remembered #13209 needs to be addressed - and it looks like we copy it to the same place for debs?
I think it's ok to leave that issue aside for now since it out-of-scope of the target spec. We can follow up on it.
Only a little scared of the package update process if we're moving things around. Would definitely like to get us CI around this first.
Soak Test Results
Baseline: 18432ef398d5c3179db96edf2a5b7824cdedadda Comparison: 8762da1a8257a54a6c776318f5ea6a9c143bc49c Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_to_http_acks | 327.23KiB | 1.79 | 84.86% | 17.82MiB | 7.71MiB | 161.17KiB | 0 | 0.432649 | 18.13MiB | 7.73MiB | 161.33KiB | 0 | 0.426067 | True | True |
| http_pipelines_blackhole_acks | 9.73KiB | 0.81 | 99.95% | 1.17MiB | 112.23KiB | 2.28KiB | 0 | 0.0935807 | 1.18MiB | 77.58KiB | 1.58KiB | 0 | 0.0641694 | False | False |
| splunk_hec_route_s3 | 128.08KiB | 0.66 | 94.99% | 18.94MiB | 2.26MiB | 47.02KiB | 0 | 0.119215 | 19.07MiB | 2.17MiB | 45.39KiB | 0 | 0.113795 | False | False |
| datadog_agent_remap_blackhole_acks | 221.05KiB | 0.34 | 84.65% | 64.27MiB | 5.72MiB | 119.07KiB | 0 | 0.0889275 | 64.49MiB | 4.73MiB | 98.99KiB | 0 | 0.0733488 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 15.2KiB | 0.06 | 80.87% | 23.82MiB | 466.16KiB | 9.51KiB | 0 | 0.0191057 | 23.84MiB | 328.13KiB | 6.7KiB | 0 | 0.0134404 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 9.35KiB | 0.04 | 29.48% | 23.74MiB | 876.61KiB | 17.83KiB | 0 | 0.0360462 | 23.75MiB | 840.85KiB | 17.11KiB | 0 | 0.0345627 | False | False |
| splunk_hec_indexer_ack_blackhole | 6.61KiB | 0.03 | 21.33% | 23.75MiB | 857.97KiB | 17.45KiB | 0 | 0.0352719 | 23.76MiB | 839.16KiB | 17.07KiB | 0 | 0.0344894 | False | False |
| syslog_log2metric_humio_metrics | -4.82KiB | -0.04 | 28.00% | 13.25MiB | 427.5KiB | 8.72KiB | 0 | 0.0315027 | 13.24MiB | 502.78KiB | 10.25KiB | 0 | 0.0370637 | False | False |
| file_to_blackhole | -46.47KiB | -0.05 | 36.75% | 95.34MiB | 3.15MiB | 65.27KiB | 0 | 0.0330249 | 95.3MiB | 3.46MiB | 71.97KiB | 0 | 0.0363026 | False | False |
| http_to_http_json | -27.7KiB | -0.11 | 97.92% | 23.85MiB | 335.64KiB | 6.85KiB | 0 | 0.0137422 | 23.82MiB | 480.83KiB | 9.83KiB | 0 | 0.0197095 | False | False |
| fluent_elasticsearch | -134.34KiB | -0.17 | 100.00% | 79.47MiB | 53.02KiB | 1.07KiB | 0 | 0.000651359 | 79.34MiB | 1.23MiB | 25.38KiB | 0 | 0.0155309 | False | False |
| http_pipelines_blackhole | -2.8KiB | -0.17 | 85.34% | 1.63MiB | 22.89KiB | 479.32B | 0 | 0.0137408 | 1.62MiB | 91.53KiB | 1.87KiB | 0 | 0.0550293 | False | False |
| syslog_regex_logs2metric_ddmetrics | -27.82KiB | -0.22 | 94.01% | 12.33MiB | 492.65KiB | 10.04KiB | 0 | 0.0390108 | 12.3MiB | 532.09KiB | 10.84KiB | 0 | 0.0422262 | False | False |
| http_to_http_noack | -69.63KiB | -0.29 | 99.86% | 23.84MiB | 405.2KiB | 8.28KiB | 0 | 0.0165972 | 23.77MiB | 991.89KiB | 20.21KiB | 0 | 0.0407447 | False | False |
| syslog_splunk_hec_logs | -50.01KiB | -0.31 | 99.46% | 15.75MiB | 696.8KiB | 14.18KiB | 0 | 0.0431847 | 15.7MiB | 539.44KiB | 11.01KiB | 0 | 0.0335363 | False | False |
| datadog_agent_remap_blackhole | -295.95KiB | -0.44 | 99.52% | 65.34MiB | 4.33MiB | 90.17KiB | 0 | 0.0662136 | 65.05MiB | 2.56MiB | 53.39KiB | 0 | 0.0392965 | False | False |
| syslog_humio_logs | -80.25KiB | -0.47 | 100.00% | 16.67MiB | 133.92KiB | 2.73KiB | 0 | 0.007846 | 16.59MiB | 155.44KiB | 3.18KiB | 0 | 0.00914989 | False | False |
| syslog_log2metric_splunk_hec_metrics | -112.68KiB | -0.64 | 100.00% | 17.07MiB | 898.85KiB | 18.31KiB | 0 | 0.0514154 | 16.96MiB | 982.54KiB | 20.0KiB | 0 | 0.0565674 | False | False |
| http_pipelines_no_grok_blackhole | -114.07KiB | -0.97 | 100.00% | 11.53MiB | 56.71KiB | 1.16KiB | 0 | 0.00480402 | 11.41MiB | 1.16MiB | 24.21KiB | 0 | 0.10189 | False | False |
| datadog_agent_remap_datadog_logs_acks | -1.06MiB | -1.61 | 100.00% | 66.22MiB | 3.18MiB | 66.51KiB | 0 | 0.048018 | 65.16MiB | 4.61MiB | 96.03KiB | 0 | 0.0707831 | False | False |
| datadog_agent_remap_datadog_logs | -1.41MiB | -2.12 | 100.00% | 66.74MiB | 408.45KiB | 8.36KiB | 0 | 0.00597547 | 65.32MiB | 4.63MiB | 96.35KiB | 0 | 0.0708423 | False | False |
| syslog_loki | -360.58KiB | -2.39 | 100.00% | 14.75MiB | 632.73KiB | 12.96KiB | 0 | 0.0418733 | 14.4MiB | 858.09KiB | 17.44KiB | 0 | 0.0581761 | False | False |
| http_text_to_http_json | -1003.26KiB | -2.41 | 100.00% | 40.71MiB | 739.48KiB | 15.09KiB | 0 | 0.0177362 | 39.73MiB | 690.81KiB | 14.1KiB | 0 | 0.0169774 | False | False |
| socket_to_socket_blackhole | -391.9KiB | -2.72 | 100.00% | 14.08MiB | 98.38KiB | 2.01KiB | 0 | 0.00682269 | 13.7MiB | 114.1KiB | 2.33KiB | 0 | 0.00813439 | False | False |
Soak Test Results
Baseline: d9b06441d95b4155c05662e3d2768ccbf8dc005f Comparison: b172be21cb17f75c415e4957372d64f16e14ca5e Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_pipelines_blackhole | 29.51KiB | 1.9 | 100.00% | 1.51MiB | 60.8KiB | 1.24KiB | 0 | 0.0392246 | 1.54MiB | 128.47KiB | 2.62KiB | 0 | 0.0813369 | False | False |
| http_pipelines_blackhole_acks | 14.4KiB | 1.24 | 100.00% | 1.13MiB | 101.4KiB | 2.06KiB | 0 | 0.0872662 | 1.15MiB | 83.18KiB | 1.7KiB | 0 | 0.0707072 | False | False |
| datadog_agent_remap_blackhole_acks | 513.24KiB | 0.76 | 100.00% | 66.3MiB | 4.59MiB | 95.52KiB | 0 | 0.069171 | 66.8MiB | 2.9MiB | 60.66KiB | 0 | 0.0433788 | False | False |
| http_to_http_acks | 108.67KiB | 0.6 | 34.95% | 17.83MiB | 8.15MiB | 170.47KiB | 0 | 0.457099 | 17.94MiB | 8.09MiB | 168.73KiB | 0 | 0.450984 | True | True |
| socket_to_socket_blackhole | 27.53KiB | 0.2 | 97.39% | 13.3MiB | 445.02KiB | 9.08KiB | 0 | 0.0326613 | 13.33MiB | 411.11KiB | 8.39KiB | 0 | 0.0301121 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 21.17KiB | 0.09 | 60.61% | 23.74MiB | 903.44KiB | 18.37KiB | 0 | 0.0371592 | 23.76MiB | 820.9KiB | 16.7KiB | 0 | 0.033735 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 11.11KiB | 0.05 | 68.57% | 23.83MiB | 430.21KiB | 8.78KiB | 0 | 0.0176294 | 23.84MiB | 327.56KiB | 6.69KiB | 0 | 0.0134168 | False | False |
| splunk_hec_indexer_ack_blackhole | 4.24KiB | 0.02 | 12.75% | 23.75MiB | 926.66KiB | 18.85KiB | 0 | 0.0381008 | 23.75MiB | 910.66KiB | 18.53KiB | 0 | 0.0374362 | False | False |
| file_to_blackhole | -124.79KiB | -0.13 | 77.13% | 95.34MiB | 2.93MiB | 60.84KiB | 0 | 0.0307744 | 95.22MiB | 4.04MiB | 83.91KiB | 0 | 0.0423879 | False | False |
| http_to_http_json | -45.17KiB | -0.18 | 99.92% | 23.85MiB | 338.92KiB | 6.92KiB | 0 | 0.0138766 | 23.8MiB | 569.19KiB | 11.61KiB | 0 | 0.0233479 | False | False |
| http_to_http_noack | -51.91KiB | -0.21 | 99.19% | 23.84MiB | 396.11KiB | 8.09KiB | 0 | 0.0162243 | 23.79MiB | 875.56KiB | 17.85KiB | 0 | 0.0359388 | False | False |
| fluent_elasticsearch | -190.77KiB | -0.23 | 99.99% | 79.47MiB | 97.62KiB | 1.97KiB | 0 | 0.00119935 | 79.29MiB | 2.32MiB | 47.65KiB | 0 | 0.0291941 | False | False |
| http_pipelines_no_grok_blackhole | -41.56KiB | -0.36 | 93.91% | 11.37MiB | 250.5KiB | 5.11KiB | 0 | 0.0215077 | 11.33MiB | 1.04MiB | 21.57KiB | 0 | 0.0913322 | False | False |
| splunk_hec_route_s3 | -100.25KiB | -0.5 | 88.28% | 19.49MiB | 2.18MiB | 45.36KiB | 0 | 0.111698 | 19.39MiB | 2.16MiB | 45.11KiB | 0 | 0.111184 | False | False |
| datadog_agent_remap_blackhole | -465.7KiB | -0.72 | 99.91% | 63.47MiB | 5.06MiB | 105.37KiB | 0 | 0.0796425 | 63.01MiB | 4.47MiB | 93.35KiB | 0 | 0.070996 | False | False |
| syslog_log2metric_humio_metrics | -103.76KiB | -0.77 | 100.00% | 13.17MiB | 326.64KiB | 6.67KiB | 0 | 0.0242196 | 13.07MiB | 586.52KiB | 11.94KiB | 0 | 0.0438266 | False | False |
| syslog_regex_logs2metric_ddmetrics | -118.72KiB | -0.95 | 100.00% | 12.15MiB | 726.11KiB | 14.8KiB | 0 | 0.0583415 | 12.04MiB | 662.36KiB | 13.51KiB | 0 | 0.0537313 | False | False |
| datadog_agent_remap_datadog_logs_acks | -662.21KiB | -1.03 | 100.00% | 62.85MiB | 3.26MiB | 68.15KiB | 0 | 0.051869 | 62.2MiB | 4.39MiB | 91.34KiB | 0 | 0.0705333 | False | False |
| datadog_agent_remap_datadog_logs | -903.66KiB | -1.35 | 100.00% | 65.42MiB | 2.48MiB | 51.97KiB | 0 | 0.0379173 | 64.54MiB | 4.83MiB | 100.57KiB | 0 | 0.0748335 | False | False |
| syslog_splunk_hec_logs | -256.17KiB | -1.49 | 100.00% | 16.82MiB | 701.09KiB | 14.28KiB | 0 | 0.0407054 | 16.57MiB | 588.09KiB | 12.0KiB | 0 | 0.0346599 | False | False |
| syslog_humio_logs | -254.38KiB | -1.49 | 100.00% | 16.69MiB | 131.35KiB | 2.68KiB | 0 | 0.00768374 | 16.44MiB | 142.84KiB | 2.93KiB | 0 | 0.0084824 | False | False |
| syslog_log2metric_splunk_hec_metrics | -304.95KiB | -1.73 | 100.00% | 17.25MiB | 884.42KiB | 18.03KiB | 0 | 0.0500455 | 16.96MiB | 938.93KiB | 19.13KiB | 0 | 0.0540632 | False | False |
| http_text_to_http_json | -837.97KiB | -1.98 | 100.00% | 41.24MiB | 711.0KiB | 14.51KiB | 0 | 0.0168313 | 40.43MiB | 702.0KiB | 14.33KiB | 0 | 0.0169546 | False | False |
| syslog_loki | -384.1KiB | -2.61 | 100.00% | 14.4MiB | 857.78KiB | 17.55KiB | 0 | 0.0581649 | 14.02MiB | 884.14KiB | 17.97KiB | 0 | 0.0615557 | False | False |
Soak Test Results
Baseline: 1a47fd6f989a4f6595b6e36eee2db260a25b6540 Comparison: 12aa75c02f582e12c11b754d45037e1005b58614 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_pipelines_blackhole | 29.81KiB | 1.84 | 100.00% | 1.58MiB | 109.28KiB | 2.23KiB | 0 | 0.0675137 | 1.61MiB | 134.49KiB | 2.74KiB | 0 | 0.0815862 | False | False |
| datadog_agent_remap_blackhole_acks | 424.28KiB | 0.63 | 99.87% | 65.52MiB | 5.22MiB | 108.72KiB | 0 | 0.0796928 | 65.94MiB | 3.55MiB | 74.21KiB | 0 | 0.053755 | False | False |
| datadog_agent_remap_blackhole | 300.27KiB | 0.44 | 98.50% | 66.15MiB | 4.42MiB | 92.1KiB | 0 | 0.0668144 | 66.44MiB | 3.94MiB | 82.07KiB | 0 | 0.0592371 | False | False |
| splunk_hec_route_s3 | 40.33KiB | 0.21 | 45.13% | 18.91MiB | 2.31MiB | 48.12KiB | 0 | 0.122154 | 18.95MiB | 2.25MiB | 46.97KiB | 0 | 0.118614 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 24.99KiB | 0.1 | 96.10% | 23.81MiB | 492.0KiB | 10.03KiB | 0 | 0.0201731 | 23.84MiB | 331.43KiB | 6.77KiB | 0 | 0.0135755 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 12.02KiB | 0.05 | 37.27% | 23.74MiB | 883.98KiB | 17.97KiB | 0 | 0.036354 | 23.75MiB | 836.58KiB | 17.02KiB | 0 | 0.0343876 | False | False |
| splunk_hec_indexer_ack_blackhole | -12.58KiB | -0.05 | 38.93% | 23.76MiB | 826.17KiB | 16.81KiB | 0 | 0.0339501 | 23.75MiB | 891.09KiB | 18.12KiB | 0 | 0.0366369 | False | False |
| socket_to_socket_blackhole | -9.69KiB | -0.07 | 86.22% | 13.62MiB | 224.88KiB | 4.59KiB | 0 | 0.016123 | 13.61MiB | 227.43KiB | 4.64KiB | 0 | 0.0163167 | False | False |
| file_to_blackhole | -80.26KiB | -0.08 | 64.06% | 95.36MiB | 2.69MiB | 55.74KiB | 0 | 0.0281926 | 95.28MiB | 3.25MiB | 67.52KiB | 0 | 0.0340923 | False | False |
| http_to_http_json | -31.51KiB | -0.13 | 98.67% | 23.84MiB | 346.75KiB | 7.08KiB | 0 | 0.0141986 | 23.81MiB | 517.21KiB | 10.57KiB | 0 | 0.0212055 | False | False |
| fluent_elasticsearch | -237.37KiB | -0.29 | 100.00% | 79.47MiB | 53.65KiB | 1.08KiB | 0 | 0.000659082 | 79.24MiB | 1.93MiB | 39.74KiB | 0 | 0.0243825 | False | False |
| http_pipelines_blackhole_acks | -3.61KiB | -0.31 | 79.93% | 1.14MiB | 108.89KiB | 2.22KiB | 0 | 0.0930799 | 1.14MiB | 85.36KiB | 1.74KiB | 0 | 0.0731908 | False | False |
| http_to_http_noack | -85.45KiB | -0.35 | 99.99% | 23.85MiB | 247.72KiB | 5.07KiB | 0 | 0.010143 | 23.76MiB | 1021.6KiB | 20.81KiB | 0 | 0.0419772 | False | False |
| syslog_log2metric_splunk_hec_metrics | -89.85KiB | -0.5 | 99.92% | 17.4MiB | 978.38KiB | 19.95KiB | 0 | 0.0548861 | 17.32MiB | 883.91KiB | 17.99KiB | 0 | 0.0498374 | False | False |
| syslog_log2metric_humio_metrics | -99.99KiB | -0.72 | 100.00% | 13.51MiB | 253.68KiB | 5.18KiB | 0 | 0.018327 | 13.42MiB | 454.82KiB | 9.27KiB | 0 | 0.0330974 | False | False |
| http_pipelines_no_grok_blackhole | -94.88KiB | -0.82 | 99.99% | 11.24MiB | 233.47KiB | 4.77KiB | 0 | 0.0202761 | 11.15MiB | 1.16MiB | 24.19KiB | 0 | 0.104212 | False | False |
| http_text_to_http_json | -421.8KiB | -1.02 | 100.00% | 40.36MiB | 1.41MiB | 29.54KiB | 0 | 0.0349992 | 39.95MiB | 1.17MiB | 24.38KiB | 0 | 0.0291768 | False | False |
| syslog_regex_logs2metric_ddmetrics | -134.18KiB | -1.04 | 100.00% | 12.62MiB | 622.39KiB | 12.68KiB | 0 | 0.0481694 | 12.48MiB | 537.53KiB | 10.96KiB | 0 | 0.0420389 | False | False |
| syslog_splunk_hec_logs | -222.54KiB | -1.33 | 100.00% | 16.29MiB | 854.67KiB | 17.41KiB | 0 | 0.0512324 | 16.07MiB | 766.81KiB | 15.64KiB | 0 | 0.0465868 | False | False |
| http_to_http_acks | -290.34KiB | -1.57 | 78.93% | 18.08MiB | 7.8MiB | 163.03KiB | 0 | 0.431353 | 17.79MiB | 7.91MiB | 164.96KiB | 0 | 0.444249 | True | True |
| syslog_humio_logs | -274.5KiB | -1.6 | 100.00% | 16.71MiB | 463.68KiB | 9.47KiB | 0 | 0.027098 | 16.44MiB | 459.21KiB | 9.4KiB | 0 | 0.0272744 | False | False |
| datadog_agent_remap_datadog_logs | -1.18MiB | -1.8 | 100.00% | 65.58MiB | 1.97MiB | 41.32KiB | 0 | 0.0300368 | 64.4MiB | 4.74MiB | 98.62KiB | 0 | 0.0735256 | False | False |
| syslog_loki | -583.97KiB | -3.73 | 100.00% | 15.29MiB | 675.83KiB | 13.84KiB | 0 | 0.043144 | 14.72MiB | 879.37KiB | 17.88KiB | 0 | 0.0583124 | False | False |
| datadog_agent_remap_datadog_logs_acks | -2.5MiB | -3.76 | 100.00% | 66.52MiB | 3.53MiB | 73.72KiB | 0 | 0.0530266 | 64.02MiB | 4.93MiB | 102.61KiB | 0 | 0.0769707 | False | False |
Soak Test Results
Baseline: ebf2e0a661a4db36a810b019607ad92a51bc2d75 Comparison: cea64d23c40b6fb059d87597de20677ae605f00d Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole_acks | 1.27MiB | 1.94 | 100.00% | 65.37MiB | 5.68MiB | 118.15KiB | 0 | 0.0867966 | 66.64MiB | 4.27MiB | 89.34KiB | 0 | 0.0640562 | False | False |
| datadog_agent_remap_datadog_logs_acks | 1.12MiB | 1.79 | 100.00% | 62.54MiB | 3.56MiB | 74.36KiB | 0 | 0.056874 | 63.66MiB | 4.93MiB | 102.71KiB | 0 | 0.0774957 | False | False |
| datadog_agent_remap_blackhole | 887.65KiB | 1.41 | 100.00% | 61.63MiB | 4.09MiB | 85.44KiB | 0 | 0.0664224 | 62.5MiB | 3.69MiB | 77.05KiB | 0 | 0.059084 | False | False |
| http_pipelines_blackhole | 13.38KiB | 0.82 | 100.00% | 1.58MiB | 62.13KiB | 1.27KiB | 0 | 0.038291 | 1.6MiB | 118.33KiB | 2.41KiB | 0 | 0.0723292 | False | False |
| http_text_to_http_json | 206.61KiB | 0.51 | 100.00% | 39.63MiB | 1.51MiB | 31.53KiB | 0 | 0.0380561 | 39.83MiB | 1.43MiB | 29.87KiB | 0 | 0.035869 | False | False |
| datadog_agent_remap_datadog_logs | 132.75KiB | 0.21 | 70.78% | 62.68MiB | 2.98MiB | 62.53KiB | 0 | 0.047596 | 62.81MiB | 5.25MiB | 109.41KiB | 0 | 0.0836359 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 27.09KiB | 0.11 | 96.30% | 23.81MiB | 546.09KiB | 11.14KiB | 0 | 0.0223927 | 23.84MiB | 326.34KiB | 6.66KiB | 0 | 0.013367 | False | False |
| splunk_hec_indexer_ack_blackhole | 12.88KiB | 0.05 | 40.46% | 23.76MiB | 862.44KiB | 17.55KiB | 0 | 0.0354465 | 23.77MiB | 822.9KiB | 16.75KiB | 0 | 0.0338034 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 9.19KiB | 0.04 | 29.05% | 23.74MiB | 875.35KiB | 17.8KiB | 0 | 0.0359934 | 23.75MiB | 839.48KiB | 17.08KiB | 0 | 0.0345056 | False | False |
| http_pipelines_blackhole_acks | -649.83B | -0.05 | 14.49% | 1.14MiB | 127.57KiB | 2.6KiB | 0 | 0.109488 | 1.14MiB | 113.42KiB | 2.31KiB | 0 | 0.0973993 | False | False |
| file_to_blackhole | -65.71KiB | -0.07 | 49.23% | 95.36MiB | 3.15MiB | 65.25KiB | 0 | 0.0330025 | 95.3MiB | 3.59MiB | 74.7KiB | 0 | 0.0376961 | False | False |
| http_to_http_json | -29.36KiB | -0.12 | 98.31% | 23.85MiB | 337.26KiB | 6.88KiB | 0 | 0.0138091 | 23.82MiB | 497.71KiB | 10.17KiB | 0 | 0.0204028 | False | False |
| fluent_elasticsearch | -224.38KiB | -0.28 | 100.00% | 79.47MiB | 52.15KiB | 1.05KiB | 0 | 0.000640704 | 79.25MiB | 2.24MiB | 46.11KiB | 0 | 0.0282705 | False | False |
| syslog_log2metric_humio_metrics | -44.12KiB | -0.33 | 99.85% | 13.23MiB | 366.73KiB | 7.48KiB | 0 | 0.0270551 | 13.19MiB | 574.25KiB | 11.69KiB | 0 | 0.0425034 | False | False |
| splunk_hec_route_s3 | -78.59KiB | -0.4 | 77.50% | 19.38MiB | 2.22MiB | 46.18KiB | 0 | 0.114447 | 19.3MiB | 2.17MiB | 45.4KiB | 0 | 0.112529 | False | False |
| http_to_http_noack | -117.82KiB | -0.48 | 100.00% | 23.84MiB | 259.45KiB | 5.31KiB | 0 | 0.0106245 | 23.73MiB | 1.17MiB | 24.31KiB | 0 | 0.0491408 | False | False |
| http_pipelines_no_grok_blackhole | -81.93KiB | -0.72 | 99.94% | 11.09MiB | 263.31KiB | 5.38KiB | 0 | 0.0231712 | 11.01MiB | 1.12MiB | 23.36KiB | 0 | 0.10183 | False | False |
| http_to_http_acks | -238.08KiB | -1.28 | 67.44% | 18.13MiB | 8.23MiB | 171.93KiB | 0 | 0.453472 | 17.9MiB | 8.17MiB | 170.58KiB | 0 | 0.45623 | True | True |
| socket_to_socket_blackhole | -300.34KiB | -2.12 | 100.00% | 13.86MiB | 294.9KiB | 6.02KiB | 0 | 0.0207669 | 13.57MiB | 326.3KiB | 6.66KiB | 0 | 0.0234749 | False | False |
| syslog_loki | -513.86KiB | -3.26 | 100.00% | 15.38MiB | 263.65KiB | 5.4KiB | 0 | 0.0167342 | 14.88MiB | 738.58KiB | 15.01KiB | 0 | 0.0484604 | False | False |
| syslog_regex_logs2metric_ddmetrics | -576.92KiB | -4.33 | 100.00% | 13.01MiB | 667.27KiB | 13.59KiB | 0 | 0.0500848 | 12.44MiB | 512.32KiB | 10.44KiB | 0 | 0.0401953 | False | False |
| syslog_splunk_hec_logs | -907.97KiB | -5.08 | 100.00% | 17.46MiB | 800.16KiB | 16.28KiB | 0 | 0.0447364 | 16.58MiB | 563.73KiB | 11.51KiB | 0 | 0.0332035 | False | False |
| syslog_humio_logs | -1.11MiB | -6.45 | 100.00% | 17.15MiB | 124.3KiB | 2.54KiB | 0 | 0.00707442 | 16.05MiB | 127.52KiB | 2.61KiB | 0 | 0.00775839 | False | False |
| syslog_log2metric_splunk_hec_metrics | -1.35MiB | -7.27 | 100.00% | 18.59MiB | 701.6KiB | 14.3KiB | 0 | 0.0368405 | 17.24MiB | 788.81KiB | 16.06KiB | 0 | 0.0446652 | False | False |
Soak Test Results
Baseline: 4611307463ced2d069e806666178a3977265a3bc Comparison: 4127c3fb701bda83e9daf1a922d025786603e40e Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_to_http_acks | 811.71KiB | 4.53 | 99.92% | 17.49MiB | 8.04MiB | 168.01KiB | 0 | 0.459318 | 18.28MiB | 8.36MiB | 174.64KiB | 0 | 0.457332 | True | True |
| syslog_log2metric_splunk_hec_metrics | 638.5KiB | 3.5 | 100.00% | 17.79MiB | 520.17KiB | 10.6KiB | 0 | 0.028544 | 18.42MiB | 830.22KiB | 16.89KiB | 0 | 0.0440155 | False | False |
| syslog_humio_logs | 555.25KiB | 3.32 | 100.00% | 16.36MiB | 279.26KiB | 5.7KiB | 0 | 0.0166709 | 16.9MiB | 286.79KiB | 5.87KiB | 0 | 0.0165709 | False | False |
| syslog_splunk_hec_logs | 472.25KiB | 2.79 | 100.00% | 16.53MiB | 684.01KiB | 13.93KiB | 0 | 0.0404086 | 16.99MiB | 653.47KiB | 13.32KiB | 0 | 0.0375563 | False | False |
| syslog_regex_logs2metric_ddmetrics | 228.73KiB | 1.83 | 100.00% | 12.24MiB | 566.03KiB | 11.54KiB | 0 | 0.045162 | 12.46MiB | 583.36KiB | 11.9KiB | 0 | 0.0457103 | False | False |
| syslog_loki | 203.01KiB | 1.34 | 100.00% | 14.75MiB | 325.97KiB | 6.68KiB | 0 | 0.0215773 | 14.95MiB | 754.8KiB | 15.34KiB | 0 | 0.049301 | False | False |
| datadog_agent_remap_blackhole_acks | 154.01KiB | 0.24 | 69.92% | 63.89MiB | 5.64MiB | 117.43KiB | 0 | 0.0882391 | 64.04MiB | 4.37MiB | 91.41KiB | 0 | 0.0682141 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 33.17KiB | 0.14 | 98.43% | 23.8MiB | 584.38KiB | 11.92KiB | 0 | 0.0239686 | 23.84MiB | 332.9KiB | 6.8KiB | 0 | 0.0136357 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 9.05KiB | 0.04 | 29.17% | 23.75MiB | 862.21KiB | 17.53KiB | 0 | 0.0354464 | 23.76MiB | 818.96KiB | 16.66KiB | 0 | 0.0336557 | False | False |
| enterprise_http_to_http | -2.66KiB | -0.01 | 28.63% | 23.85MiB | 248.32KiB | 5.07KiB | 0 | 0.0101671 | 23.84MiB | 253.77KiB | 5.19KiB | 0 | 0.0103911 | False | False |
| file_to_blackhole | -66.14KiB | -0.07 | 48.63% | 95.34MiB | 3.07MiB | 63.57KiB | 0 | 0.0321599 | 95.28MiB | 3.79MiB | 78.82KiB | 0 | 0.0397834 | False | False |
| splunk_hec_indexer_ack_blackhole | -20.07KiB | -0.08 | 55.41% | 23.76MiB | 863.12KiB | 17.56KiB | 0 | 0.0354691 | 23.74MiB | 963.86KiB | 19.6KiB | 0 | 0.0396415 | False | False |
| http_to_http_json | -39.18KiB | -0.16 | 99.72% | 23.85MiB | 341.59KiB | 6.97KiB | 0 | 0.0139851 | 23.81MiB | 542.29KiB | 11.07KiB | 0 | 0.0222379 | False | False |
| http_pipelines_blackhole_acks | -2.09KiB | -0.17 | 58.21% | 1.19MiB | 104.27KiB | 2.12KiB | 0 | 0.0857557 | 1.19MiB | 72.32KiB | 1.47KiB | 0 | 0.0595808 | False | False |
| fluent_elasticsearch | -185.13KiB | -0.23 | 100.00% | 79.47MiB | 53.18KiB | 1.08KiB | 0 | 0.000653361 | 79.29MiB | 1.63MiB | 33.63KiB | 0 | 0.0206039 | False | False |
| datadog_agent_remap_blackhole | -198.92KiB | -0.31 | 92.68% | 63.65MiB | 4.08MiB | 85.02KiB | 0 | 0.0640596 | 63.46MiB | 3.42MiB | 71.34KiB | 0 | 0.0538976 | False | False |
| http_pipelines_no_grok_blackhole | -47.43KiB | -0.41 | 95.95% | 11.22MiB | 49.62KiB | 1.01KiB | 0 | 0.0043184 | 11.17MiB | 1.11MiB | 23.12KiB | 0 | 0.0993747 | False | False |
| datadog_agent_remap_datadog_logs_acks | -305.54KiB | -0.47 | 99.34% | 63.04MiB | 3.03MiB | 63.38KiB | 0 | 0.048073 | 62.74MiB | 4.46MiB | 92.82KiB | 0 | 0.0710558 | False | False |
| http_to_http_noack | -148.78KiB | -0.61 | 100.00% | 23.85MiB | 248.7KiB | 5.09KiB | 0 | 0.0101829 | 23.7MiB | 1.31MiB | 27.2KiB | 0 | 0.0550917 | False | False |
| splunk_hec_route_s3 | -153.19KiB | -0.78 | 98.21% | 19.11MiB | 2.24MiB | 46.56KiB | 0 | 0.116964 | 18.96MiB | 2.15MiB | 44.9KiB | 0 | 0.11314 | False | False |
| datadog_agent_remap_datadog_logs | -522.29KiB | -0.82 | 100.00% | 62.45MiB | 1.79MiB | 37.42KiB | 0 | 0.0285859 | 61.94MiB | 4.41MiB | 91.8KiB | 0 | 0.0711563 | False | False |
| http_pipelines_blackhole | -15.48KiB | -0.93 | 100.00% | 1.63MiB | 33.13KiB | 693.3B | 0 | 0.0198433 | 1.62MiB | 109.1KiB | 2.22KiB | 0 | 0.0659511 | False | False |
| syslog_log2metric_humio_metrics | -209.99KiB | -1.55 | 100.00% | 13.26MiB | 318.77KiB | 6.5KiB | 0 | 0.0234786 | 13.05MiB | 564.95KiB | 11.5KiB | 0 | 0.0422654 | False | False |
| http_text_to_http_json | -1.55MiB | -3.94 | 100.00% | 39.35MiB | 735.25KiB | 15.01KiB | 0 | 0.0182449 | 37.8MiB | 832.83KiB | 17.01KiB | 0 | 0.0215137 | False | False |
| socket_to_socket_blackhole | -920.27KiB | -3.95 | 100.00% | 22.75MiB | 654.5KiB | 13.36KiB | 0 | 0.0280897 | 21.85MiB | 593.16KiB | 12.11KiB | 0 | 0.0265041 | False | False |
Soak Test Results
Baseline: ba1e1c404f1278316cc55931cf4d3d620f5a536c Comparison: 971ab7c608f1a5ad9783f1b3f5335f7eb97b5885 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole | 1.97MiB | 3.22 | 100.00% | 61.3MiB | 5.44MiB | 113.35KiB | 0 | 0.0887499 | 63.27MiB | 3.35MiB | 69.98KiB | 0 | 0.0529712 | False | False |
| splunk_hec_route_s3 | 325.08KiB | 1.71 | 100.00% | 18.61MiB | 2.42MiB | 50.48KiB | 0 | 0.130234 | 18.93MiB | 2.17MiB | 45.49KiB | 0 | 0.114849 | False | False |
| http_pipelines_blackhole_acks | 17.72KiB | 1.41 | 100.00% | 1.22MiB | 117.44KiB | 2.39KiB | 0 | 0.0936962 | 1.24MiB | 54.15KiB | 1.1KiB | 0 | 0.0425968 | False | False |
| datadog_agent_remap_datadog_logs_acks | 272.47KiB | 0.42 | 97.98% | 63.28MiB | 3.36MiB | 70.16KiB | 0 | 0.0530243 | 63.55MiB | 4.52MiB | 93.98KiB | 0 | 0.0710317 | False | False |
| datadog_agent_remap_blackhole_acks | 261.95KiB | 0.4 | 97.14% | 63.62MiB | 4.4MiB | 91.57KiB | 0 | 0.0691029 | 63.88MiB | 3.69MiB | 76.98KiB | 0 | 0.0576883 | False | False |
| syslog_regex_logs2metric_ddmetrics | 47.2KiB | 0.38 | 99.75% | 12.1MiB | 543.18KiB | 11.07KiB | 0 | 0.0438324 | 12.15MiB | 540.59KiB | 11.02KiB | 0 | 0.0434581 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 19.7KiB | 0.08 | 89.82% | 23.82MiB | 486.45KiB | 9.93KiB | 0 | 0.0199401 | 23.84MiB | 333.77KiB | 6.81KiB | 0 | 0.0136706 | False | False |
| enterprise_http_to_http | 1.26KiB | 0.01 | 13.77% | 23.84MiB | 251.61KiB | 5.14KiB | 0 | 0.0103028 | 23.85MiB | 250.73KiB | 5.13KiB | 0 | 0.0102662 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -4.65KiB | -0.02 | 14.23% | 23.76MiB | 896.52KiB | 18.24KiB | 0 | 0.0368465 | 23.75MiB | 905.95KiB | 18.42KiB | 0 | 0.0372414 | False | False |
| splunk_hec_indexer_ack_blackhole | -13.89KiB | -0.06 | 40.86% | 23.75MiB | 878.39KiB | 17.87KiB | 0 | 0.036113 | 23.73MiB | 921.23KiB | 18.73KiB | 0 | 0.037896 | False | False |
| file_to_blackhole | -60.92KiB | -0.06 | 49.24% | 95.35MiB | 2.85MiB | 58.99KiB | 0 | 0.0298359 | 95.29MiB | 3.39MiB | 70.51KiB | 0 | 0.0355611 | False | False |
| http_to_http_json | -35.75KiB | -0.15 | 99.37% | 23.84MiB | 352.62KiB | 7.2KiB | 0 | 0.0144401 | 23.81MiB | 535.02KiB | 10.92KiB | 0 | 0.0219418 | False | False |
| http_text_to_http_json | -76.8KiB | -0.18 | 99.99% | 40.89MiB | 676.51KiB | 13.81KiB | 0 | 0.0161529 | 40.82MiB | 705.24KiB | 14.4KiB | 0 | 0.0168697 | False | False |
| syslog_log2metric_splunk_hec_metrics | -38.58KiB | -0.22 | 94.13% | 17.52MiB | 631.57KiB | 12.87KiB | 0 | 0.0351917 | 17.48MiB | 777.56KiB | 15.83KiB | 0 | 0.0434195 | False | False |
| http_to_http_noack | -77.74KiB | -0.32 | 99.99% | 23.85MiB | 255.45KiB | 5.22KiB | 0 | 0.0104594 | 23.77MiB | 971.79KiB | 19.8KiB | 0 | 0.0399174 | False | False |
| http_pipelines_blackhole | -5.87KiB | -0.34 | 99.23% | 1.7MiB | 26.35KiB | 551.25B | 0 | 0.0151712 | 1.69MiB | 104.72KiB | 2.13KiB | 0 | 0.0604957 | False | False |
| syslog_log2metric_humio_metrics | -48.85KiB | -0.37 | 99.89% | 13.03MiB | 471.99KiB | 9.64KiB | 0 | 0.0353787 | 12.98MiB | 561.43KiB | 11.43KiB | 0 | 0.0422377 | False | False |
| http_to_http_acks | -70.25KiB | -0.4 | 24.05% | 17.32MiB | 7.87MiB | 164.43KiB | 0 | 0.453932 | 17.25MiB | 7.66MiB | 160.01KiB | 0 | 0.443924 | True | True |
| fluent_elasticsearch | -394.44KiB | -0.48 | 100.00% | 79.47MiB | 54.02KiB | 1.09KiB | 0 | 0.000663695 | 79.09MiB | 4.57MiB | 93.96KiB | 0 | 0.0577897 | False | False |
| datadog_agent_remap_datadog_logs | -530.02KiB | -0.82 | 100.00% | 62.8MiB | 1.45MiB | 30.37KiB | 0 | 0.0230746 | 62.28MiB | 4.14MiB | 86.23KiB | 0 | 0.0664258 | False | False |
| syslog_splunk_hec_logs | -139.64KiB | -0.84 | 100.00% | 16.14MiB | 888.67KiB | 18.07KiB | 0 | 0.0537469 | 16.01MiB | 753.1KiB | 15.37KiB | 0 | 0.0459357 | False | False |
| syslog_loki | -126.15KiB | -0.85 | 100.00% | 14.56MiB | 492.97KiB | 10.09KiB | 0 | 0.0330674 | 14.43MiB | 796.98KiB | 16.2KiB | 0 | 0.0539167 | False | False |
| http_pipelines_no_grok_blackhole | -117.68KiB | -1.04 | 100.00% | 11.09MiB | 263.09KiB | 5.37KiB | 0 | 0.0231622 | 10.98MiB | 1.03MiB | 21.49KiB | 0 | 0.0939686 | False | False |
| syslog_humio_logs | -413.18KiB | -2.46 | 100.00% | 16.41MiB | 241.28KiB | 4.93KiB | 0 | 0.0143558 | 16.01MiB | 662.56KiB | 13.57KiB | 0 | 0.0404151 | False | False |
| socket_to_socket_blackhole | -915.23KiB | -3.76 | 100.00% | 23.8MiB | 668.55KiB | 13.65KiB | 0 | 0.0274292 | 22.9MiB | 878.01KiB | 17.93KiB | 0 | 0.0374285 | False | False |