vector
vector copied to clipboard
feat(prometheus): add auth implementation to Prometheus Exporter
Resolves https://github.com/vectordotdev/vector/issues/11041
In this PR I introduce auth functionality to Prometheus exporter. It can be used for protecting prometheus metrics from unauthorized access.
Implementation details
Auth implementation
Current HTTP server for Prometheus Exporter is implemented on Hyper. Hyper is a low-level library for HTTP so it doesn't have built-in auth middleware. There are two ways: implement auth in Vector codebase from the scratch or use more advanced HTTP server library (like tower-http, axum, warp.). I have implemented more simple way with implementing auth just a part of already existing HTTP handler for hyper.
However, I suggest you later in all data source migrate to something more high-level abstraction. It allows us to reuse more existing functionality (like built-in auth middlewares).
Testing
I have tested locally the following scenarios:
- Default behaviour without auth - works as before
- Specify auth in config file and try to get access without credentials - Unauthorized result
- Access with Basic creds - Ok
- Access with Bearer token - Ok
Deploy Preview for vector-project canceled.
Name | Link |
---|---|
Latest commit | b498b114260e1991232be340753a1e3a9076240e |
Latest deploy log | https://app.netlify.com/sites/vector-project/deploys/632cf088cbccea0008b1f5c3 |
@jszwedko would be great, if you review it :)
Soak Test Results
Baseline: d498040a770ae2bb5c9d25efce62acadcb17ee57 Comparison: 927f1bb8bcca19acb8010fd3d1eba491aa360fd6 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
syslog_splunk_hec_logs | 204.68KiB | 1.25 | 100.00% | 16.01MiB | 767.13KiB | 15.6KiB | 0 | 0.0467913 | 16.21MiB | 633.68KiB | 12.92KiB | 0 | 0.0381745 | False | False |
syslog_log2metric_splunk_hec_metrics | 210.57KiB | 1.2 | 100.00% | 17.11MiB | 672.92KiB | 13.7KiB | 0 | 0.03839 | 17.32MiB | 802.79KiB | 16.34KiB | 0 | 0.0452552 | False | False |
syslog_loki | 155.45KiB | 1.07 | 100.00% | 14.15MiB | 580.78KiB | 11.88KiB | 0 | 0.040086 | 14.3MiB | 764.35KiB | 15.54KiB | 0 | 0.052196 | False | False |
http_pipelines_blackhole_acks | 8.79KiB | 0.71 | 99.97% | 1.22MiB | 103.37KiB | 2.1KiB | 0 | 0.0829717 | 1.23MiB | 60.59KiB | 1.24KiB | 0 | 0.0482932 | False | False |
syslog_humio_logs | 99.84KiB | 0.6 | 100.00% | 16.22MiB | 137.33KiB | 2.8KiB | 0 | 0.00826426 | 16.32MiB | 235.53KiB | 4.82KiB | 0 | 0.014089 | False | False |
splunk_hec_indexer_ack_blackhole | 8.14KiB | 0.03 | 25.48% | 23.75MiB | 886.14KiB | 18.02KiB | 0 | 0.036429 | 23.76MiB | 855.85KiB | 17.42KiB | 0 | 0.0351722 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 6.72KiB | 0.03 | 48.14% | 23.83MiB | 381.07KiB | 7.79KiB | 0 | 0.015612 | 23.84MiB | 338.01KiB | 6.9KiB | 0 | 0.0138442 | False | False |
enterprise_http_to_http | -1.38KiB | -0.01 | 15.08% | 23.85MiB | 251.5KiB | 5.13KiB | 0 | 0.0102973 | 23.85MiB | 251.72KiB | 5.15KiB | 0 | 0.0103068 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -1.29KiB | -0.01 | 4.48% | 23.77MiB | 794.91KiB | 16.18KiB | 0 | 0.0326546 | 23.77MiB | 799.59KiB | 16.27KiB | 0 | 0.0328484 | False | False |
splunk_hec_route_s3 | -8.51KiB | -0.05 | 9.96% | 17.7MiB | 2.33MiB | 48.55KiB | 0 | 0.131599 | 17.69MiB | 2.28MiB | 47.6KiB | 0 | 0.128664 | False | False |
file_to_blackhole | -75.98KiB | -0.08 | 55.71% | 95.34MiB | 3.1MiB | 64.31KiB | 0 | 0.0325318 | 95.27MiB | 3.62MiB | 75.29KiB | 0 | 0.0380304 | False | False |
http_pipelines_no_grok_blackhole | -14.95KiB | -0.14 | 52.75% | 10.48MiB | 74.25KiB | 1.52KiB | 0 | 0.00692023 | 10.46MiB | 1020.44KiB | 20.76KiB | 0 | 0.0952377 | False | False |
http_to_http_json | -46.66KiB | -0.19 | 99.95% | 23.85MiB | 329.85KiB | 6.73KiB | 0 | 0.0135046 | 23.8MiB | 569.91KiB | 11.63KiB | 0 | 0.0233778 | False | False |
fluent_elasticsearch | -188.03KiB | -0.23 | 100.00% | 79.47MiB | 52.88KiB | 1.07KiB | 0 | 0.000649671 | 79.29MiB | 1.71MiB | 35.11KiB | 0 | 0.0215246 | False | False |
http_to_http_noack | -70.53KiB | -0.29 | 99.88% | 23.84MiB | 402.21KiB | 8.22KiB | 0 | 0.0164741 | 23.77MiB | 985.48KiB | 20.08KiB | 0 | 0.0404808 | False | False |
datadog_agent_remap_blackhole | -261.76KiB | -0.42 | 98.22% | 61.12MiB | 4.27MiB | 88.94KiB | 0 | 0.0698714 | 60.87MiB | 3.14MiB | 65.46KiB | 0 | 0.0515492 | False | False |
syslog_regex_logs2metric_ddmetrics | -58.25KiB | -0.45 | 99.96% | 12.76MiB | 579.7KiB | 11.81KiB | 0 | 0.0443426 | 12.71MiB | 564.24KiB | 11.5KiB | 0 | 0.0433531 | False | False |
http_to_http_acks | -92.09KiB | -0.52 | 30.46% | 17.37MiB | 7.87MiB | 164.53KiB | 0 | 0.453012 | 17.28MiB | 8.06MiB | 168.09KiB | 0 | 0.466381 | True | True |
http_text_to_http_json | -259.23KiB | -0.67 | 100.00% | 37.96MiB | 1.05MiB | 21.89KiB | 0 | 0.0275739 | 37.71MiB | 1.07MiB | 22.4KiB | 0 | 0.0284055 | False | False |
datadog_agent_remap_datadog_logs_acks | -592.23KiB | -0.94 | 100.00% | 61.36MiB | 2.94MiB | 61.49KiB | 0 | 0.0479191 | 60.78MiB | 4.3MiB | 89.43KiB | 0 | 0.0706773 | False | False |
datadog_agent_remap_blackhole_acks | -627.99KiB | -1.06 | 100.00% | 57.96MiB | 3.95MiB | 82.28KiB | 0 | 0.0681439 | 57.35MiB | 3.53MiB | 73.74KiB | 0 | 0.0615721 | False | False |
http_pipelines_blackhole | -35.14KiB | -2.19 | 100.00% | 1.56MiB | 178.82KiB | 3.65KiB | 0 | 0.111575 | 1.53MiB | 217.04KiB | 4.43KiB | 0 | 0.138461 | False | False |
socket_to_socket_blackhole | -681.59KiB | -2.8 | 100.00% | 23.75MiB | 297.75KiB | 6.08KiB | 0 | 0.0122385 | 23.09MiB | 453.88KiB | 9.27KiB | 0 | 0.0191937 | False | False |
syslog_log2metric_humio_metrics | -364.7KiB | -2.94 | 100.00% | 12.11MiB | 577.17KiB | 11.78KiB | 0 | 0.0465474 | 11.75MiB | 948.47KiB | 19.3KiB | 0 | 0.0788103 | False | False |
datadog_agent_remap_datadog_logs | -2.65MiB | -4.47 | 100.00% | 59.41MiB | 3.26MiB | 68.39KiB | 0 | 0.0548992 | 56.76MiB | 6.11MiB | 127.31KiB | 0 | 0.107666 | False | False |
Soak Test Results
Baseline: d498040a770ae2bb5c9d25efce62acadcb17ee57 Comparison: 3ea743080d495bc4cb76dc96c1a19ce6273fa0e9 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_pipelines_blackhole_acks | 9.02KiB | 0.73 | 99.71% | 1.21MiB | 119.85KiB | 2.44KiB | 0 | 0.0968383 | 1.22MiB | 87.85KiB | 1.79KiB | 0 | 0.0704658 | False | False |
syslog_splunk_hec_logs | 82.98KiB | 0.5 | 99.99% | 16.25MiB | 747.53KiB | 15.21KiB | 0 | 0.0449183 | 16.33MiB | 671.46KiB | 13.68KiB | 0 | 0.0401472 | False | False |
syslog_log2metric_humio_metrics | 29.92KiB | 0.23 | 99.43% | 12.81MiB | 203.94KiB | 4.17KiB | 0 | 0.0155387 | 12.84MiB | 489.54KiB | 9.97KiB | 0 | 0.0372147 | False | False |
http_pipelines_blackhole | 2.88KiB | 0.18 | 77.41% | 1.6MiB | 26.2KiB | 547.99B | 0 | 0.0159904 | 1.6MiB | 113.53KiB | 2.31KiB | 0 | 0.0691656 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 26.11KiB | 0.11 | 95.20% | 23.81MiB | 552.34KiB | 11.26KiB | 0 | 0.0226492 | 23.84MiB | 336.93KiB | 6.88KiB | 0 | 0.0138014 | False | False |
splunk_hec_to_splunk_hec_logs_acks | 2.56KiB | 0.01 | 8.55% | 23.76MiB | 837.33KiB | 17.04KiB | 0 | 0.0344123 | 23.76MiB | 822.23KiB | 16.73KiB | 0 | 0.0337884 | False | False |
enterprise_http_to_http | 719.85B | 0 | 7.61% | 23.85MiB | 253.35KiB | 5.17KiB | 0 | 0.0103737 | 23.85MiB | 256.24KiB | 5.24KiB | 0 | 0.0104913 | False | False |
splunk_hec_indexer_ack_blackhole | -3.36KiB | -0.01 | 10.51% | 23.75MiB | 881.43KiB | 17.93KiB | 0 | 0.0362368 | 23.75MiB | 885.37KiB | 18.01KiB | 0 | 0.0364035 | False | False |
file_to_blackhole | -33.56KiB | -0.03 | 28.83% | 95.34MiB | 2.98MiB | 61.85KiB | 0 | 0.0312861 | 95.31MiB | 3.19MiB | 66.48KiB | 0 | 0.0335147 | False | False |
http_to_http_json | -42.55KiB | -0.17 | 99.88% | 23.85MiB | 331.4KiB | 6.77KiB | 0 | 0.0135681 | 23.81MiB | 553.0KiB | 11.29KiB | 0 | 0.0226804 | False | False |
fluent_elasticsearch | -188.15KiB | -0.23 | 100.00% | 79.47MiB | 53.89KiB | 1.09KiB | 0 | 0.000662054 | 79.29MiB | 1.72MiB | 35.36KiB | 0 | 0.0216513 | False | False |
syslog_loki | -45.91KiB | -0.31 | 99.42% | 14.44MiB | 367.15KiB | 7.51KiB | 0 | 0.0248197 | 14.4MiB | 730.65KiB | 14.85KiB | 0 | 0.0495469 | False | False |
syslog_humio_logs | -57.31KiB | -0.34 | 100.00% | 16.68MiB | 264.71KiB | 5.4KiB | 0 | 0.0154971 | 16.62MiB | 164.66KiB | 3.37KiB | 0 | 0.00967212 | False | False |
http_to_http_noack | -88.22KiB | -0.36 | 99.89% | 23.82MiB | 610.82KiB | 12.48KiB | 0 | 0.0250368 | 23.73MiB | 1.15MiB | 24.03KiB | 0 | 0.0485552 | False | False |
splunk_hec_route_s3 | -108.23KiB | -0.56 | 90.68% | 18.9MiB | 2.21MiB | 45.94KiB | 0 | 0.116646 | 18.8MiB | 2.16MiB | 45.21KiB | 0 | 0.114874 | False | False |
syslog_regex_logs2metric_ddmetrics | -84.9KiB | -0.66 | 100.00% | 12.66MiB | 612.53KiB | 12.48KiB | 0 | 0.0472526 | 12.57MiB | 555.72KiB | 11.33KiB | 0 | 0.0431531 | False | False |
datadog_agent_remap_datadog_logs_acks | -423.88KiB | -0.67 | 99.98% | 61.59MiB | 3.25MiB | 67.91KiB | 0 | 0.0527642 | 61.17MiB | 4.32MiB | 90.0KiB | 0 | 0.0706648 | False | False |
http_text_to_http_json | -298.48KiB | -0.75 | 100.00% | 38.63MiB | 817.07KiB | 16.68KiB | 0 | 0.0206492 | 38.34MiB | 907.96KiB | 18.53KiB | 0 | 0.0231207 | False | False |
syslog_log2metric_splunk_hec_metrics | -195.67KiB | -1.09 | 100.00% | 17.48MiB | 581.9KiB | 11.86KiB | 0 | 0.0324993 | 17.29MiB | 746.82KiB | 15.22KiB | 0 | 0.0421712 | False | False |
datadog_agent_remap_blackhole | -715.7KiB | -1.2 | 100.00% | 58.36MiB | 4.08MiB | 84.85KiB | 0 | 0.0698147 | 57.66MiB | 2.81MiB | 58.59KiB | 0 | 0.0486898 | False | False |
http_pipelines_no_grok_blackhole | -136.75KiB | -1.21 | 100.00% | 10.99MiB | 46.04KiB | 963.03B | 0 | 0.00408998 | 10.86MiB | 1.11MiB | 23.05KiB | 0 | 0.101974 | False | False |
http_to_http_acks | -298.7KiB | -1.65 | 79.97% | 17.7MiB | 7.89MiB | 164.92KiB | 0 | 0.445588 | 17.41MiB | 7.9MiB | 164.87KiB | 0 | 0.453944 | True | True |
datadog_agent_remap_datadog_logs | -1.04MiB | -1.7 | 100.00% | 61.24MiB | 811.92KiB | 16.63KiB | 0 | 0.0129439 | 60.2MiB | 4.18MiB | 87.12KiB | 0 | 0.0694986 | False | False |
datadog_agent_remap_blackhole_acks | -2.51MiB | -4.12 | 100.00% | 60.98MiB | 4.23MiB | 88.01KiB | 0 | 0.0692792 | 58.47MiB | 3.3MiB | 68.88KiB | 0 | 0.0564009 | False | False |
socket_to_socket_blackhole | -981.58KiB | -4.21 | 100.00% | 22.79MiB | 1.5MiB | 31.36KiB | 0 | 0.0658181 | 21.83MiB | 1.7MiB | 35.58KiB | 0 | 0.0779531 | False | False |
I left a couple of small notes, but, also, do you think we could add some tests for this behavior? You should be able to copy and modify the ones already in src/sinks/prometheus/exporter.rs.
Yes, I definitely want add some tests. I just waited for the feedback from your side. Right now seems that in general you agree with this implementation, so I can start work on tests.
I left a couple of small notes, but, also, do you think we could add some tests for this behavior? You should be able to copy and modify the ones already in src/sinks/prometheus/exporter.rs.
Yes, I definitely want add some tests. I just waited for the feedback from your side. Right now seems that in general you agree with this implementation, so I can start work on tests.
👍 I agree with your assessment in the PR description. It would be nice to rely on middleware for this in the future, but what you have makes sense for this sink which is not currently using a HTTP framework like axum or warp.
@jszwedko by the way, CI says that for user/password I need to use special type instead of usual String
. Just want to clarify - should I fix it too or it is just a false positive?
Soak Test Results
Baseline: 50f37f5b1a5910d51d698773d23026270b07b7e8 Comparison: a7067a1b8b296c30e8cfe110c429278dc711cf87 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
syslog_log2metric_splunk_hec_metrics | 684.3KiB | 3.87 | 100.00% | 17.29MiB | 947.09KiB | 19.3KiB | 0 | 0.053492 | 17.95MiB | 767.6KiB | 15.63KiB | 0 | 0.0417408 | False | False |
syslog_splunk_hec_logs | 574.92KiB | 3.52 | 100.00% | 15.94MiB | 825.03KiB | 16.78KiB | 0 | 0.0505468 | 16.5MiB | 669.4KiB | 13.64KiB | 0 | 0.0396156 | False | False |
syslog_humio_logs | 506.45KiB | 3.08 | 100.00% | 16.03MiB | 740.8KiB | 15.12KiB | 0 | 0.0451093 | 16.53MiB | 569.55KiB | 11.66KiB | 0 | 0.0336434 | False | False |
syslog_regex_logs2metric_ddmetrics | 274.07KiB | 2.17 | 100.00% | 12.33MiB | 555.18KiB | 11.31KiB | 0 | 0.0439455 | 12.6MiB | 599.01KiB | 12.2KiB | 0 | 0.0464081 | False | False |
datadog_agent_remap_blackhole | 1.06MiB | 1.85 | 100.00% | 57.27MiB | 4.8MiB | 100.06KiB | 0 | 0.0838156 | 58.33MiB | 4.16MiB | 86.7KiB | 0 | 0.0712761 | False | False |
socket_to_socket_blackhole | 266.22KiB | 1.17 | 100.00% | 22.3MiB | 975.33KiB | 19.91KiB | 0 | 0.0426947 | 22.56MiB | 1016.95KiB | 20.76KiB | 0 | 0.0440038 | False | False |
http_pipelines_blackhole_acks | 12.39KiB | 0.99 | 100.00% | 1.22MiB | 99.17KiB | 2.02KiB | 0 | 0.0794603 | 1.23MiB | 66.73KiB | 1.36KiB | 0 | 0.0529454 | False | False |
splunk_hec_indexer_ack_blackhole | 16.66KiB | 0.07 | 47.54% | 23.74MiB | 939.66KiB | 19.11KiB | 0 | 0.0386423 | 23.76MiB | 879.07KiB | 17.89KiB | 0 | 0.0361261 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 10.49KiB | 0.04 | 65.74% | 23.83MiB | 425.04KiB | 8.68KiB | 0 | 0.0174159 | 23.84MiB | 335.0KiB | 6.84KiB | 0 | 0.0137209 | False | False |
enterprise_http_to_http | 1.48KiB | 0.01 | 16.04% | 23.84MiB | 252.51KiB | 5.15KiB | 0 | 0.0103399 | 23.85MiB | 252.05KiB | 5.16KiB | 0 | 0.0103201 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -18.76KiB | -0.08 | 55.04% | 23.77MiB | 830.57KiB | 16.9KiB | 0 | 0.0341209 | 23.75MiB | 892.66KiB | 18.15KiB | 0 | 0.0366999 | False | False |
datadog_agent_remap_blackhole_acks | -65.03KiB | -0.11 | 48.60% | 58.2MiB | 3.98MiB | 82.87KiB | 0 | 0.0683796 | 58.13MiB | 2.65MiB | 55.34KiB | 0 | 0.0455499 | False | False |
http_to_http_json | -35.47KiB | -0.15 | 99.34% | 23.84MiB | 349.34KiB | 7.13KiB | 0 | 0.0143058 | 23.81MiB | 535.21KiB | 10.92KiB | 0 | 0.021949 | False | False |
file_to_blackhole | -162.04KiB | -0.17 | 77.24% | 95.31MiB | 4.01MiB | 83.01KiB | 0 | 0.0420468 | 95.15MiB | 5.08MiB | 105.54KiB | 0 | 0.0534068 | False | False |
http_pipelines_no_grok_blackhole | -21.32KiB | -0.2 | 64.44% | 10.33MiB | 403.41KiB | 8.23KiB | 0 | 0.0381142 | 10.31MiB | 1.04MiB | 21.56KiB | 0 | 0.100351 | False | False |
fluent_elasticsearch | -173.23KiB | -0.21 | 99.99% | 79.47MiB | 54.13KiB | 1.1KiB | 0 | 0.000664978 | 79.3MiB | 2.1MiB | 43.14KiB | 0 | 0.0264413 | False | False |
http_pipelines_blackhole | -3.53KiB | -0.21 | 82.27% | 1.66MiB | 20.38KiB | 426.54B | 0 | 0.0120079 | 1.65MiB | 126.89KiB | 2.58KiB | 0 | 0.0749306 | False | False |
http_to_http_noack | -57.9KiB | -0.24 | 99.46% | 23.84MiB | 416.9KiB | 8.53KiB | 0 | 0.0170773 | 23.78MiB | 931.8KiB | 18.99KiB | 0 | 0.0382596 | False | False |
splunk_hec_route_s3 | -144.75KiB | -0.76 | 97.09% | 18.7MiB | 2.31MiB | 48.01KiB | 0 | 0.123286 | 18.56MiB | 2.18MiB | 45.73KiB | 0 | 0.117709 | False | False |
datadog_agent_remap_datadog_logs_acks | -515.01KiB | -0.81 | 100.00% | 62.02MiB | 3.42MiB | 71.34KiB | 0 | 0.055065 | 61.52MiB | 4.38MiB | 91.2KiB | 0 | 0.071203 | False | False |
http_to_http_acks | -155.76KiB | -0.88 | 48.70% | 17.2MiB | 8.12MiB | 169.71KiB | 0 | 0.471921 | 17.05MiB | 8.01MiB | 166.97KiB | 0 | 0.469707 | True | True |
syslog_loki | -159.9KiB | -1.08 | 100.00% | 14.39MiB | 554.6KiB | 11.34KiB | 0 | 0.0376169 | 14.24MiB | 791.91KiB | 16.1KiB | 0 | 0.0543018 | False | False |
datadog_agent_remap_datadog_logs | -1.33MiB | -2.19 | 100.00% | 60.66MiB | 2.21MiB | 46.42KiB | 0 | 0.0364745 | 59.34MiB | 4.79MiB | 99.63KiB | 0 | 0.0806296 | False | False |
syslog_log2metric_humio_metrics | -321.89KiB | -2.51 | 100.00% | 12.54MiB | 626.66KiB | 12.79KiB | 0 | 0.0487973 | 12.22MiB | 730.27KiB | 14.88KiB | 0 | 0.058328 | False | False |
http_text_to_http_json | -1.13MiB | -2.92 | 100.00% | 38.68MiB | 1.12MiB | 23.46KiB | 0 | 0.0290106 | 37.55MiB | 1.11MiB | 23.27KiB | 0 | 0.0296475 | False | False |
Soak Test Results
Baseline: 7b26f65c6bde9fe824ea15948fa0147cf026db5e Comparison: 9dc93b67f9a2bcffb9480431f53639ee62376b0d Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
splunk_hec_route_s3 | 341.67KiB | 1.83 | 100.00% | 18.27MiB | 2.39MiB | 49.71KiB | 0 | 0.130665 | 18.6MiB | 2.28MiB | 47.55KiB | 0 | 0.122322 | False | False |
syslog_splunk_hec_logs | 267.98KiB | 1.63 | 100.00% | 16.05MiB | 812.37KiB | 16.54KiB | 0 | 0.0494125 | 16.31MiB | 747.42KiB | 15.23KiB | 0 | 0.0447324 | False | False |
syslog_log2metric_splunk_hec_metrics | 255.6KiB | 1.45 | 100.00% | 17.26MiB | 833.52KiB | 16.98KiB | 0 | 0.0471592 | 17.51MiB | 934.77KiB | 19.03KiB | 0 | 0.0521333 | False | False |
syslog_humio_logs | 225.21KiB | 1.35 | 100.00% | 16.32MiB | 371.53KiB | 7.58KiB | 0 | 0.0222264 | 16.54MiB | 310.7KiB | 6.36KiB | 0 | 0.0183399 | False | False |
syslog_loki | 198.76KiB | 1.35 | 100.00% | 14.43MiB | 511.36KiB | 10.47KiB | 0 | 0.0346011 | 14.62MiB | 780.31KiB | 15.86KiB | 0 | 0.0520995 | False | False |
syslog_regex_logs2metric_ddmetrics | 169.17KiB | 1.32 | 100.00% | 12.53MiB | 681.35KiB | 13.86KiB | 0 | 0.0530732 | 12.7MiB | 466.71KiB | 9.52KiB | 0 | 0.0358809 | False | False |
http_pipelines_blackhole_acks | 13.88KiB | 1.17 | 100.00% | 1.16MiB | 121.92KiB | 2.48KiB | 0 | 0.103003 | 1.17MiB | 107.51KiB | 2.19KiB | 0 | 0.0897745 | False | False |
syslog_log2metric_humio_metrics | 133.28KiB | 1.06 | 100.00% | 12.33MiB | 791.42KiB | 16.15KiB | 0 | 0.0626436 | 12.47MiB | 877.04KiB | 17.85KiB | 0 | 0.0686964 | False | False |
splunk_hec_to_splunk_hec_logs_acks | 23.9KiB | 0.1 | 69.60% | 23.75MiB | 855.49KiB | 17.4KiB | 0 | 0.0351665 | 23.78MiB | 757.4KiB | 15.42KiB | 0 | 0.0311035 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 11.38KiB | 0.05 | 69.71% | 23.83MiB | 426.28KiB | 8.7KiB | 0 | 0.0174681 | 23.84MiB | 333.56KiB | 6.81KiB | 0 | 0.0136625 | False | False |
splunk_hec_indexer_ack_blackhole | 1.49KiB | 0.01 | 4.24% | 23.73MiB | 980.28KiB | 19.93KiB | 0 | 0.0403269 | 23.74MiB | 972.55KiB | 19.77KiB | 0 | 0.0400061 | False | False |
enterprise_http_to_http | -632.94B | -0 | 6.63% | 23.85MiB | 253.34KiB | 5.17KiB | 0 | 0.0103728 | 23.85MiB | 260.98KiB | 5.34KiB | 0 | 0.0106857 | False | False |
file_to_blackhole | -67.79KiB | -0.07 | 49.74% | 95.34MiB | 3.32MiB | 68.85KiB | 0 | 0.0348346 | 95.27MiB | 3.56MiB | 74.04KiB | 0 | 0.0373582 | False | False |
http_to_http_json | -30.2KiB | -0.12 | 98.69% | 23.85MiB | 333.54KiB | 6.81KiB | 0 | 0.0136566 | 23.82MiB | 493.3KiB | 10.08KiB | 0 | 0.0202226 | False | False |
datadog_agent_remap_blackhole | -78.19KiB | -0.13 | 53.54% | 60.5MiB | 4.27MiB | 88.92KiB | 0 | 0.0705669 | 60.42MiB | 2.84MiB | 59.36KiB | 0 | 0.047071 | False | False |
http_to_http_acks | -42.47KiB | -0.24 | 14.52% | 17.25MiB | 7.88MiB | 164.71KiB | 0 | 0.456723 | 17.21MiB | 7.82MiB | 163.36KiB | 0 | 0.454556 | True | True |
fluent_elasticsearch | -192.11KiB | -0.24 | 100.00% | 79.47MiB | 54.19KiB | 1.1KiB | 0 | 0.000665705 | 79.29MiB | 1.71MiB | 35.22KiB | 0 | 0.0215749 | False | False |
http_to_http_noack | -76.11KiB | -0.31 | 99.83% | 23.83MiB | 511.31KiB | 10.45KiB | 0 | 0.0209516 | 23.75MiB | 1.05MiB | 21.83KiB | 0 | 0.0440672 | False | False |
datadog_agent_remap_blackhole_acks | -473.53KiB | -0.75 | 100.00% | 61.7MiB | 3.94MiB | 82.13KiB | 0 | 0.0638734 | 61.24MiB | 2.98MiB | 62.37KiB | 0 | 0.0486987 | False | False |
http_pipelines_no_grok_blackhole | -85.25KiB | -0.78 | 99.91% | 10.66MiB | 507.01KiB | 10.35KiB | 0 | 0.046418 | 10.58MiB | 1.12MiB | 23.36KiB | 0 | 0.105986 | False | False |
socket_to_socket_blackhole | -425.3KiB | -1.84 | 100.00% | 22.58MiB | 1.3MiB | 27.25KiB | 0 | 0.0577108 | 22.17MiB | 1.12MiB | 23.49KiB | 0 | 0.0506595 | False | False |
datadog_agent_remap_datadog_logs_acks | -1.5MiB | -2.37 | 100.00% | 63.17MiB | 2.9MiB | 60.59KiB | 0 | 0.0458374 | 61.67MiB | 4.38MiB | 91.16KiB | 0 | 0.0709946 | False | False |
http_pipelines_blackhole | -49.18KiB | -2.92 | 100.00% | 1.65MiB | 98.48KiB | 2.01KiB | 0 | 0.0583823 | 1.6MiB | 151.05KiB | 3.08KiB | 0 | 0.0922439 | False | False |
http_text_to_http_json | -1.17MiB | -3.03 | 100.00% | 38.58MiB | 1.63MiB | 34.14KiB | 0 | 0.0423224 | 37.41MiB | 1.36MiB | 28.35KiB | 0 | 0.0362322 | False | False |
datadog_agent_remap_datadog_logs | -2.14MiB | -3.5 | 100.00% | 61.3MiB | 2.0MiB | 41.92KiB | 0 | 0.0326272 | 59.15MiB | 4.13MiB | 86.04KiB | 0 | 0.0698194 | False | False |
I have synced my branch with the upstream, added SensitiveString
using and added const HeaderValue
usage.
Soak Test Results
Baseline: ab3ca33efecdc91eff89fb9c0bb3a1ab81cd2762 Comparison: 46c0b5e190f908b3fb27424fe57b2e0f4e7ec504 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_to_http_acks | 157.02KiB | 0.89 | 51.05% | 17.28MiB | 7.75MiB | 162.15KiB | 0 | 0.448533 | 17.43MiB | 7.63MiB | 159.15KiB | 0 | 0.437366 | True | True |
syslog_regex_logs2metric_ddmetrics | 47.11KiB | 0.38 | 99.95% | 11.95MiB | 479.35KiB | 9.76KiB | 0 | 0.0391573 | 12.0MiB | 452.56KiB | 9.22KiB | 0 | 0.0368264 | False | False |
syslog_humio_logs | 57.83KiB | 0.35 | 100.00% | 16.11MiB | 204.51KiB | 4.18KiB | 0 | 0.0123931 | 16.17MiB | 206.33KiB | 4.22KiB | 0 | 0.0124594 | False | False |
http_pipelines_blackhole_acks | 1.94KiB | 0.15 | 47.00% | 1.23MiB | 114.67KiB | 2.33KiB | 0 | 0.0912648 | 1.23MiB | 98.8KiB | 2.01KiB | 0 | 0.0785142 | False | False |
http_text_to_http_json | 48.02KiB | 0.12 | 84.00% | 39.24MiB | 1.1MiB | 22.93KiB | 0 | 0.027948 | 39.28MiB | 1.21MiB | 25.34KiB | 0 | 0.0308534 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 26.12KiB | 0.11 | 95.68% | 23.81MiB | 543.2KiB | 11.08KiB | 0 | 0.0222716 | 23.84MiB | 325.18KiB | 6.64KiB | 0 | 0.0133184 | False | False |
splunk_hec_indexer_ack_blackhole | 17.2KiB | 0.07 | 51.06% | 23.75MiB | 895.32KiB | 18.22KiB | 0 | 0.0368024 | 23.77MiB | 832.78KiB | 16.95KiB | 0 | 0.0342074 | False | False |
enterprise_http_to_http | -2.38KiB | -0.01 | 25.57% | 23.85MiB | 249.81KiB | 5.1KiB | 0 | 0.0102278 | 23.84MiB | 255.76KiB | 5.23KiB | 0 | 0.0104726 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -7.72KiB | -0.03 | 23.59% | 23.75MiB | 868.4KiB | 17.66KiB | 0 | 0.0356976 | 23.74MiB | 919.0KiB | 18.69KiB | 0 | 0.0377896 | False | False |
splunk_hec_route_s3 | -26.64KiB | -0.14 | 30.79% | 18.56MiB | 2.32MiB | 48.21KiB | 0 | 0.124713 | 18.53MiB | 2.24MiB | 46.94KiB | 0 | 0.121092 | False | False |
datadog_agent_remap_blackhole_acks | -89.61KiB | -0.14 | 61.80% | 62.04MiB | 4.15MiB | 86.36KiB | 0 | 0.0668161 | 61.95MiB | 2.64MiB | 55.18KiB | 0 | 0.042553 | False | False |
file_to_blackhole | -192.53KiB | -0.2 | 82.68% | 95.32MiB | 4.13MiB | 85.45KiB | 0 | 0.04328 | 95.13MiB | 5.43MiB | 112.6KiB | 0 | 0.0570173 | False | False |
http_to_http_json | -54.47KiB | -0.22 | 99.99% | 23.85MiB | 329.28KiB | 6.72KiB | 0 | 0.0134813 | 23.79MiB | 603.28KiB | 12.3KiB | 0 | 0.0247545 | False | False |
socket_to_socket_blackhole | -53.45KiB | -0.22 | 99.57% | 23.23MiB | 600.22KiB | 12.25KiB | 0 | 0.0252303 | 23.17MiB | 693.26KiB | 14.15KiB | 0 | 0.0292073 | False | False |
datadog_agent_remap_datadog_logs_acks | -190.47KiB | -0.31 | 93.59% | 60.02MiB | 2.59MiB | 54.22KiB | 0 | 0.0431601 | 59.83MiB | 4.2MiB | 87.39KiB | 0 | 0.0701513 | False | False |
http_to_http_noack | -114.53KiB | -0.47 | 100.00% | 23.85MiB | 255.55KiB | 5.22KiB | 0 | 0.0104633 | 23.73MiB | 1.15MiB | 24.02KiB | 0 | 0.0485348 | False | False |
syslog_splunk_hec_logs | -81.02KiB | -0.49 | 99.99% | 16.13MiB | 791.83KiB | 16.1KiB | 0 | 0.0479298 | 16.05MiB | 661.7KiB | 13.5KiB | 0 | 0.0402499 | False | False |
datadog_agent_remap_datadog_logs | -390.85KiB | -0.62 | 100.00% | 61.93MiB | 1.26MiB | 26.42KiB | 0 | 0.0203531 | 61.55MiB | 4.33MiB | 90.09KiB | 0 | 0.0702771 | False | False |
fluent_elasticsearch | -536.66KiB | -0.66 | 100.00% | 79.47MiB | 52.67KiB | 1.06KiB | 0 | 0.000647103 | 78.95MiB | 5.24MiB | 107.6KiB | 0 | 0.0663603 | False | False |
syslog_log2metric_humio_metrics | -108.96KiB | -0.84 | 100.00% | 12.66MiB | 319.54KiB | 6.52KiB | 0 | 0.0246477 | 12.55MiB | 511.0KiB | 10.4KiB | 0 | 0.0397509 | False | False |
syslog_loki | -157.78KiB | -1.09 | 100.00% | 14.15MiB | 673.63KiB | 13.8KiB | 0 | 0.0464889 | 13.99MiB | 1.01MiB | 20.96KiB | 0 | 0.0719365 | False | False |
http_pipelines_no_grok_blackhole | -142.86KiB | -1.31 | 100.00% | 10.62MiB | 162.86KiB | 3.33KiB | 0 | 0.0149664 | 10.49MiB | 1.05MiB | 21.9KiB | 0 | 0.100293 | False | False |
http_pipelines_blackhole | -26.61KiB | -1.55 | 100.00% | 1.67MiB | 56.35KiB | 1.15KiB | 0 | 0.0329107 | 1.65MiB | 122.36KiB | 2.49KiB | 0 | 0.0725932 | False | False |
syslog_log2metric_splunk_hec_metrics | -553.07KiB | -3.14 | 100.00% | 17.22MiB | 1005.01KiB | 20.49KiB | 0 | 0.0569691 | 16.68MiB | 1.48MiB | 30.83KiB | 0 | 0.0885956 | False | False |
datadog_agent_remap_blackhole | -2.29MiB | -3.75 | 100.00% | 60.91MiB | 4.59MiB | 95.67KiB | 0 | 0.0754113 | 58.62MiB | 4.92MiB | 102.57KiB | 0 | 0.0838777 | False | False |
I got one more warning regarding using lazy_static
instead of const. @jszwedko according to use Vector guidelines, should I use here something like once_cell::sync::Lazy
or just left it as a const will be fine?
I guess explicit use base64;
should resolve the issue :)
Argh. @jszwedko could you please help me resolve the issue with base64
error? I canot understand what I am doing wrong. I have already checked other base64
usages in Vector codebase and didn't find any difference with my case. I can successfully build Vector binary locally (that is how I test it locally). But why it fails on CI... I do not know. As you see, explicit use base64;
didn't help. Could you locally checkout the branch and just try to build it?
Argh. @jszwedko could you please help me resolve the issue with
base64
error? I canot understand what I am doing wrong. I have already checked otherbase64
usages in Vector codebase and didn't find any difference with my case. I can successfully build Vector binary locally (that is how I test it locally). But why it fails on CI... I do not know. As you see, explicituse base64;
didn't help. Could you locally checkout the branch and just try to build it?
Doh, I'm sorry, I led you in the wrong direction. The issue is that the base64
crate is marked as optional in Cargo.toml
so must be specifically included for components that need it. I pushed db023d2ca to remedy this. This explains why you didn't see the issue locally though since it would only show up when building subsets of Vector that don't result in base64
being included.
Ah, that one was kinda tricky :) Thank you for the help!
@jszwedko I have fixed your commit: you mistakenly added dep:base64
to sources-prometheus
, not sinks-prometheus
. Now it builds fine but soak tests are failed for some reason. I am completely sure that is not caused by my changes. So maybe fix them somehow on your side or just suppress them and merge without soak tests.
Soak Test Results
Baseline: 0d94ae2b80c80e57deea9af62388766577f4b7b1 Comparison: db023d2ca75bc404502d31007bd7309c510074cb Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_text_to_http_json | 1.15MiB | 3.06 | 100.00% | 37.59MiB | 1.02MiB | 21.25KiB | 0 | 0.0270455 | 38.74MiB | 1.36MiB | 28.48KiB | 0 | 0.0351638 | False | False |
syslog_log2metric_humio_metrics | 220.57KiB | 1.76 | 100.00% | 12.23MiB | 446.94KiB | 9.13KiB | 0 | 0.0356881 | 12.44MiB | 599.87KiB | 12.21KiB | 0 | 0.0470706 | False | False |
datadog_agent_remap_datadog_logs_acks | 796.83KiB | 1.26 | 100.00% | 61.57MiB | 3.47MiB | 72.52KiB | 0 | 0.0563876 | 62.35MiB | 4.48MiB | 93.32KiB | 0 | 0.0718889 | False | False |
http_pipelines_blackhole_acks | 11.74KiB | 0.94 | 100.00% | 1.23MiB | 120.79KiB | 2.46KiB | 0 | 0.0962411 | 1.24MiB | 54.69KiB | 1.12KiB | 0 | 0.0431724 | False | False |
syslog_splunk_hec_logs | 63.48KiB | 0.39 | 99.89% | 16.06MiB | 760.5KiB | 15.47KiB | 0 | 0.0462288 | 16.12MiB | 577.71KiB | 11.78KiB | 0 | 0.0349824 | False | False |
syslog_regex_logs2metric_ddmetrics | 44.36KiB | 0.35 | 99.39% | 12.21MiB | 572.24KiB | 11.65KiB | 0 | 0.0457564 | 12.25MiB | 550.83KiB | 11.23KiB | 0 | 0.0438888 | False | False |
datadog_agent_remap_datadog_logs | 211.18KiB | 0.34 | 97.30% | 61.42MiB | 1.03MiB | 21.68KiB | 0 | 0.016828 | 61.63MiB | 4.47MiB | 92.97KiB | 0 | 0.0724439 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 38.98KiB | 0.16 | 99.43% | 23.8MiB | 604.28KiB | 12.33KiB | 0 | 0.0247878 | 23.84MiB | 334.67KiB | 6.83KiB | 0 | 0.0137062 | False | False |
enterprise_http_to_http | 3.4KiB | 0.01 | 35.69% | 23.84MiB | 253.84KiB | 5.18KiB | 0 | 0.010395 | 23.85MiB | 253.84KiB | 5.19KiB | 0 | 0.0103937 | False | False |
socket_to_socket_blackhole | -789.38B | -0 | 1.97% | 22.63MiB | 1.0MiB | 20.93KiB | 0 | 0.0442299 | 22.63MiB | 1.11MiB | 23.22KiB | 0 | 0.0490481 | False | False |
file_to_blackhole | -52.37KiB | -0.05 | 36.88% | 95.34MiB | 3.39MiB | 70.18KiB | 0 | 0.0355014 | 95.29MiB | 4.01MiB | 83.5KiB | 0 | 0.0421245 | False | False |
splunk_hec_to_splunk_hec_logs_acks | -21.08KiB | -0.09 | 58.69% | 23.78MiB | 890.3KiB | 18.12KiB | 0 | 0.0365563 | 23.76MiB | 900.12KiB | 18.31KiB | 0 | 0.0369917 | False | False |
splunk_hec_indexer_ack_blackhole | -26.14KiB | -0.11 | 69.54% | 23.77MiB | 828.47KiB | 16.87KiB | 0 | 0.0340296 | 23.74MiB | 937.63KiB | 19.07KiB | 0 | 0.0385551 | False | False |
http_to_http_json | -32.43KiB | -0.13 | 98.95% | 23.84MiB | 348.28KiB | 7.11KiB | 0 | 0.0142619 | 23.81MiB | 513.47KiB | 10.49KiB | 0 | 0.0210547 | False | False |
syslog_log2metric_splunk_hec_metrics | -31.97KiB | -0.18 | 90.47% | 17.58MiB | 590.5KiB | 12.04KiB | 0 | 0.0327907 | 17.55MiB | 732.22KiB | 14.91KiB | 0 | 0.0407332 | False | False |
http_pipelines_blackhole | -4.58KiB | -0.27 | 99.02% | 1.68MiB | 16.73KiB | 350.37B | 0 | 0.00971081 | 1.68MiB | 85.23KiB | 1.74KiB | 0 | 0.0496095 | False | False |
http_to_http_acks | -49.77KiB | -0.28 | 16.85% | 17.17MiB | 7.82MiB | 163.58KiB | 0 | 0.455576 | 17.12MiB | 8.02MiB | 167.29KiB | 0 | 0.468203 | True | True |
fluent_elasticsearch | -241.88KiB | -0.3 | 100.00% | 79.47MiB | 51.88KiB | 1.05KiB | 0 | 0.000637324 | 79.24MiB | 2.5MiB | 51.49KiB | 0 | 0.0316048 | False | False |
datadog_agent_remap_blackhole_acks | -231.25KiB | -0.37 | 94.59% | 61.4MiB | 4.67MiB | 97.23KiB | 0 | 0.0760639 | 61.17MiB | 3.37MiB | 70.37KiB | 0 | 0.0549982 | False | False |
http_to_http_noack | -95.55KiB | -0.39 | 99.99% | 23.84MiB | 404.65KiB | 8.27KiB | 0 | 0.0165745 | 23.74MiB | 1.1MiB | 22.93KiB | 0 | 0.0463036 | False | False |
syslog_humio_logs | -68.48KiB | -0.42 | 100.00% | 16.07MiB | 577.45KiB | 11.79KiB | 0 | 0.0350782 | 16.01MiB | 556.66KiB | 11.4KiB | 0 | 0.0339564 | False | False |
splunk_hec_route_s3 | -151.29KiB | -0.79 | 97.55% | 18.6MiB | 2.35MiB | 48.84KiB | 0 | 0.126154 | 18.45MiB | 2.21MiB | 46.21KiB | 0 | 0.119782 | False | False |
http_pipelines_no_grok_blackhole | -194.87KiB | -1.75 | 100.00% | 10.88MiB | 40.41KiB | 844.93B | 0 | 0.00362743 | 10.69MiB | 1.03MiB | 21.5KiB | 0 | 0.0966169 | False | False |
datadog_agent_remap_blackhole | -1.29MiB | -2.18 | 100.00% | 59.22MiB | 4.83MiB | 100.53KiB | 0 | 0.0814703 | 57.93MiB | 4.6MiB | 96.02KiB | 0 | 0.0794651 | False | False |
syslog_loki | -373.5KiB | -2.53 | 100.00% | 14.44MiB | 587.07KiB | 12.02KiB | 0 | 0.0396947 | 14.08MiB | 941.1KiB | 19.13KiB | 0 | 0.0652816 | False | False |
Doh, thanks for fixing that @zamazan4ik . I think the soak test failure was due to one of the CI runners going away. I restarted them.
Soak Test Results
Baseline: 18aead8d070976a64313939191fb9635623fc69e Comparison: b498b114260e1991232be340753a1e3a9076240e Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
http_text_to_http_json | 1.36MiB | 3.63 | 100.00% | 37.55MiB | 1.08MiB | 22.51KiB | 0 | 0.0286783 | 38.91MiB | 1.24MiB | 26.01KiB | 0 | 0.0319675 | False | False |
syslog_log2metric_humio_metrics | 165.31KiB | 1.29 | 100.00% | 12.52MiB | 436.52KiB | 8.91KiB | 0 | 0.0340332 | 12.68MiB | 535.4KiB | 10.91KiB | 0 | 0.0412111 | False | False |
datadog_agent_remap_datadog_logs_acks | 744.09KiB | 1.27 | 100.00% | 57.29MiB | 3.8MiB | 79.43KiB | 0 | 0.0663154 | 58.02MiB | 5.01MiB | 104.33KiB | 0 | 0.0863508 | False | False |
socket_to_socket_blackhole | 210.51KiB | 0.88 | 100.00% | 23.24MiB | 325.99KiB | 6.65KiB | 0 | 0.0136979 | 23.44MiB | 644.74KiB | 13.17KiB | 0 | 0.0268544 | False | False |
http_pipelines_blackhole_acks | 5.88KiB | 0.47 | 94.59% | 1.22MiB | 120.28KiB | 2.45KiB | 0 | 0.0965276 | 1.22MiB | 89.49KiB | 1.82KiB | 0 | 0.0714797 | False | False |
syslog_regex_logs2metric_ddmetrics | 51.55KiB | 0.42 | 99.99% | 11.96MiB | 529.24KiB | 10.77KiB | 0 | 0.0432118 | 12.01MiB | 388.07KiB | 7.91KiB | 0 | 0.0315528 | False | False |
datadog_agent_remap_datadog_logs | 256.17KiB | 0.41 | 99.75% | 61.29MiB | 412.69KiB | 8.45KiB | 0 | 0.00657405 | 61.54MiB | 4.05MiB | 84.27KiB | 0 | 0.0657258 | False | False |
splunk_hec_indexer_ack_blackhole | 24.39KiB | 0.1 | 64.13% | 23.74MiB | 968.77KiB | 19.7KiB | 0 | 0.0398505 | 23.76MiB | 876.14KiB | 17.83KiB | 0 | 0.0360039 | False | False |
syslog_humio_logs | 14.18KiB | 0.09 | 90.73% | 16.22MiB | 293.38KiB | 5.99KiB | 0 | 0.0176601 | 16.23MiB | 289.96KiB | 5.94KiB | 0 | 0.0174392 | False | False |
splunk_hec_to_splunk_hec_logs_acks | 14.62KiB | 0.06 | 43.19% | 23.74MiB | 915.45KiB | 18.61KiB | 0 | 0.0376569 | 23.75MiB | 865.02KiB | 17.59KiB | 0 | 0.0355609 | False | False |
splunk_hec_to_splunk_hec_logs_noack | 15.03KiB | 0.06 | 79.48% | 23.82MiB | 472.86KiB | 9.65KiB | 0 | 0.0193809 | 23.84MiB | 337.5KiB | 6.89KiB | 0 | 0.0138243 | False | False |
enterprise_http_to_http | -3.73KiB | -0.02 | 38.34% | 23.85MiB | 257.27KiB | 5.25KiB | 0 | 0.0105333 | 23.84MiB | 258.49KiB | 5.29KiB | 0 | 0.0105846 | False | False |
file_to_blackhole | -37.43KiB | -0.04 | 30.47% | 95.34MiB | 3.01MiB | 62.32KiB | 0 | 0.0315231 | 95.3MiB | 3.48MiB | 72.42KiB | 0 | 0.0365047 | False | False |
http_to_http_json | -33.76KiB | -0.14 | 99.33% | 23.85MiB | 333.54KiB | 6.81KiB | 0 | 0.013656 | 23.81MiB | 509.39KiB | 10.4KiB | 0 | 0.0208849 | False | False |
datadog_agent_remap_blackhole_acks | -151.8KiB | -0.26 | 72.75% | 57.98MiB | 5.18MiB | 107.83KiB | 0 | 0.0892917 | 57.83MiB | 4.14MiB | 86.63KiB | 0 | 0.0715175 | False | False |
syslog_log2metric_splunk_hec_metrics | -54.28KiB | -0.31 | 99.17% | 17.34MiB | 554.6KiB | 11.31KiB | 0 | 0.0312328 | 17.28MiB | 843.37KiB | 17.16KiB | 0 | 0.0476402 | False | False |
fluent_elasticsearch | -357.26KiB | -0.44 | 100.00% | 79.47MiB | 54.15KiB | 1.09KiB | 0 | 0.000665287 | 79.12MiB | 4.12MiB | 84.62KiB | 0 | 0.0520132 | False | False |
http_to_http_noack | -110.5KiB | -0.45 | 100.00% | 23.84MiB | 397.57KiB | 8.13KiB | 0 | 0.0162849 | 23.73MiB | 1.17MiB | 24.49KiB | 0 | 0.0495047 | False | False |
datadog_agent_remap_blackhole | -345.32KiB | -0.57 | 99.88% | 59.31MiB | 3.73MiB | 77.77KiB | 0 | 0.0628459 | 58.98MiB | 3.47MiB | 72.45KiB | 0 | 0.0588958 | False | False |
http_to_http_acks | -158.76KiB | -0.88 | 50.13% | 17.58MiB | 7.81MiB | 163.31KiB | 0 | 0.444113 | 17.43MiB | 8.07MiB | 168.47KiB | 0 | 0.462903 | True | True |
http_pipelines_blackhole | -17.23KiB | -1 | 100.00% | 1.69MiB | 47.81KiB | 999.64B | 0 | 0.0276374 | 1.67MiB | 129.41KiB | 2.64KiB | 0 | 0.0755547 | False | False |
http_pipelines_no_grok_blackhole | -154.91KiB | -1.41 | 100.00% | 10.73MiB | 289.23KiB | 5.9KiB | 0 | 0.0263147 | 10.58MiB | 1.0MiB | 20.89KiB | 0 | 0.0947739 | False | False |
splunk_hec_route_s3 | -289.95KiB | -1.5 | 100.00% | 18.85MiB | 2.26MiB | 46.95KiB | 0 | 0.11966 | 18.56MiB | 2.21MiB | 46.29KiB | 0 | 0.119247 | False | False |
syslog_loki | -353.34KiB | -2.43 | 100.00% | 14.17MiB | 717.88KiB | 14.68KiB | 0 | 0.0494483 | 13.83MiB | 916.65KiB | 18.63KiB | 0 | 0.064715 | False | False |
syslog_splunk_hec_logs | -538.04KiB | -3.35 | 100.00% | 15.69MiB | 1.16MiB | 24.15KiB | 0 | 0.0738282 | 15.16MiB | 1.35MiB | 28.13KiB | 0 | 0.0888735 | False | False |
Merged! Thanks for this contribution!