vectordotdev
docs(gcp_pubsub source): use generated docs for config
Closes #15902 Epic #14998
Stop! Don't review yet - I've just found a whole bunch of cue errors...
Deploy Preview for vrl-playground canceled.
| Name | Link |
|---|---|
| Latest commit | e3b480d0be52f429d26051eb82e1f199c07e8783 |
| Latest deploy log | https://app.netlify.com/sites/vrl-playground/deploys/63c12bfca1b619000c1a1b99 |
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
Deploy Preview for vector-project canceled.
| Name | Link |
|---|---|
| Latest commit | e3b480d0be52f429d26051eb82e1f199c07e8783 |
| Latest deploy log | https://app.netlify.com/sites/vector-project/deploys/63c12bfc36da950009f51a22 |
Regression Test Results
Run ID: 05d21cd1-a1f0-4644-97b8-14e3c67566c2
Baseline: 4af6d78b88661d874547f336a740b845aacd9334
Comparison: dc2b7bc1e4daac9f02d5df4d57baf50982ca67e1
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| syslog_log2metric_humio_metrics | -340.85KiB/CPU-s | -5.51 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_regex_logs2metric_ddmetrics | 134.56KiB/CPU-s | 3.66 | 100.00% | 3.59MiB/CPU-s | 448.64KiB/CPU-s | 5.79KiB/CPU-s | 0.0 | 0.121964 | 3.72MiB/CPU-s | 448.52KiB/CPU-s | 5.79KiB/CPU-s | 0.0 | 0.117629 | True | False |
| otlp_http_to_blackhole | 50.4KiB/CPU-s | 3.34 | 100.00% | 1.47MiB/CPU-s | 135.93KiB/CPU-s | 1.75KiB/CPU-s | 0.0 | 0.09011 | 1.52MiB/CPU-s | 121.08KiB/CPU-s | 1.56KiB/CPU-s | 0.0 | 0.077667 | False | False |
| datadog_agent_remap_blackhole_acks | 781.43KiB/CPU-s | 2.49 | 100.00% | 30.61MiB/CPU-s | 1.32MiB/CPU-s | 17.43KiB/CPU-s | 0.0 | 0.043055 | 31.38MiB/CPU-s | 1.33MiB/CPU-s | 17.57KiB/CPU-s | 0.0 | 0.042354 | False | False |
| splunk_hec_route_s3 | 148.06KiB/CPU-s | 1.24 | 100.00% | 11.7MiB/CPU-s | 680.47KiB/CPU-s | 8.78KiB/CPU-s | 0.0 | 0.056815 | 11.84MiB/CPU-s | 524.0KiB/CPU-s | 6.76KiB/CPU-s | 0.0 | 0.043217 | False | False |
| datadog_agent_remap_blackhole | 367.71KiB/CPU-s | 1.18 | 100.00% | 30.52MiB/CPU-s | 1.18MiB/CPU-s | 15.55KiB/CPU-s | 0.0 | 0.038529 | 30.88MiB/CPU-s | 1.3MiB/CPU-s | 17.21KiB/CPU-s | 0.0 | 0.042179 | False | False |
| syslog_loki | 71.24KiB/CPU-s | 0.78 | 100.00% | 8.87MiB/CPU-s | 251.9KiB/CPU-s | 3.25KiB/CPU-s | 0.0 | 0.027743 | 8.94MiB/CPU-s | 113.16KiB/CPU-s | 1.46KiB/CPU-s | 0.0 | 0.012366 | False | False |
| datadog_agent_remap_datadog_logs_acks | 262.2KiB/CPU-s | 0.74 | 100.00% | 34.38MiB/CPU-s | 1011.45KiB/CPU-s | 13.05KiB/CPU-s | 0.0 | 0.028727 | 34.64MiB/CPU-s | 746.69KiB/CPU-s | 9.64KiB/CPU-s | 0.0 | 0.021051 | False | False |
| socket_to_socket_blackhole | 95.02KiB/CPU-s | 0.69 | 100.00% | 13.5MiB/CPU-s | 211.6KiB/CPU-s | 2.73KiB/CPU-s | 0.0 | 0.015305 | 13.59MiB/CPU-s | 171.27KiB/CPU-s | 2.21KiB/CPU-s | 0.0 | 0.012303 | False | False |
| http_to_http_acks | 25.14KiB/CPU-s | 0.47 | 37.31% | 5.21MiB/CPU-s | 2.75MiB/CPU-s | 36.28KiB/CPU-s | 0.0 | 0.52693 | 5.24MiB/CPU-s | 2.79MiB/CPU-s | 36.85KiB/CPU-s | 0.0 | 0.532519 | True | False |
| syslog_humio_logs | 22.9KiB/CPU-s | 0.25 | 99.97% | 8.92MiB/CPU-s | 416.69KiB/CPU-s | 5.38KiB/CPU-s | 0.0 | 0.045624 | 8.94MiB/CPU-s | 255.94KiB/CPU-s | 3.3KiB/CPU-s | 0.0 | 0.027954 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 3.62KiB/CPU-s | 0.03 | 66.60% | 13.62MiB/CPU-s | 218.88KiB/CPU-s | 2.82KiB/CPU-s | 0.0 | 0.01569 | 13.63MiB/CPU-s | 190.81KiB/CPU-s | 2.46KiB/CPU-s | 0.0 | 0.013675 | False | False |
| enterprise_http_to_http | 4.29KiB/CPU-s | 0.03 | 63.73% | 13.62MiB/CPU-s | 285.31KiB/CPU-s | 3.68KiB/CPU-s | 0.0 | 0.020462 | 13.62MiB/CPU-s | 227.81KiB/CPU-s | 2.94KiB/CPU-s | 0.0 | 0.016333 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 1.42KiB/CPU-s | 0.01 | 17.13% | 13.61MiB/CPU-s | 370.48KiB/CPU-s | 4.78KiB/CPU-s | 0.0 | 0.026573 | 13.62MiB/CPU-s | 350.39KiB/CPU-s | 4.52KiB/CPU-s | 0.0 | 0.025129 | False | False |
| splunk_hec_indexer_ack_blackhole | -460.03B/CPU-s | -0.0 | 7.66% | 13.62MiB/CPU-s | 254.8KiB/CPU-s | 3.29KiB/CPU-s | 0.0 | 0.018272 | 13.62MiB/CPU-s | 257.33KiB/CPU-s | 3.32KiB/CPU-s | 0.0 | 0.018454 | False | False |
| http_to_http_json | 187.06B/CPU-s | 0.0 | 3.76% | 13.62MiB/CPU-s | 212.95KiB/CPU-s | 2.75KiB/CPU-s | 0.0 | 0.015265 | 13.62MiB/CPU-s | 211.81KiB/CPU-s | 2.73KiB/CPU-s | 0.0 | 0.015183 | False | False |
| fluent_elasticsearch | -12.25B/CPU-s | -0.0 | 1.76% | 45.41MiB/CPU-s | 29.8KiB/CPU-s | 389.48B/CPU-s | 0.0 | 0.000641 | 45.41MiB/CPU-s | 30.15KiB/CPU-s | 394.02B/CPU-s | 0.0 | 0.000648 | False | False |
| http_to_http_noack | -2.14KiB/CPU-s | -0.02 | 26.92% | 13.61MiB/CPU-s | 330.59KiB/CPU-s | 4.27KiB/CPU-s | 0.0 | 0.023716 | 13.61MiB/CPU-s | 352.32KiB/CPU-s | 4.55KiB/CPU-s | 0.0 | 0.025278 | False | False |
| file_to_blackhole | -16.27KiB/CPU-s | -0.03 | 54.88% | 54.5MiB/CPU-s | 1.1MiB/CPU-s | 14.52KiB/CPU-s | 0.0 | 0.020167 | 54.48MiB/CPU-s | 1.21MiB/CPU-s | 15.99KiB/CPU-s | 0.0 | 0.022241 | False | False |
| otlp_grpc_to_blackhole | -770.8B/CPU-s | -0.07 | 62.39% | 1.03MiB/CPU-s | 37.95KiB/CPU-s | 501.74B/CPU-s | 0.0 | 0.035926 | 1.03MiB/CPU-s | 53.87KiB/CPU-s | 711.75B/CPU-s | 0.0 | 0.051029 | False | False |
| datadog_agent_remap_datadog_logs | -79.49KiB/CPU-s | -0.22 | 99.96% | 34.72MiB/CPU-s | 1.13MiB/CPU-s | 14.92KiB/CPU-s | 0.0 | 0.032542 | 34.64MiB/CPU-s | 1.27MiB/CPU-s | 16.75KiB/CPU-s | 0.0 | 0.036602 | False | False |
| syslog_log2metric_splunk_hec_metrics | -93.69KiB/CPU-s | -0.97 | 100.00% | 9.41MiB/CPU-s | 437.04KiB/CPU-s | 5.64KiB/CPU-s | 0.0 | 0.045345 | 9.32MiB/CPU-s | 334.48KiB/CPU-s | 4.32KiB/CPU-s | 0.0 | 0.035045 | False | False |
| http_text_to_http_json | -284.46KiB/CPU-s | -1.08 | 100.00% | 25.64MiB/CPU-s | 611.83KiB/CPU-s | 7.9KiB/CPU-s | 0.0 | 0.023305 | 25.36MiB/CPU-s | 913.52KiB/CPU-s | 11.79KiB/CPU-s | 0.0 | 0.035178 | False | False |
| syslog_splunk_hec_logs | -111.87KiB/CPU-s | -1.21 | 100.00% | 9.03MiB/CPU-s | 228.18KiB/CPU-s | 2.95KiB/CPU-s | 0.0 | 0.024684 | 8.92MiB/CPU-s | 327.41KiB/CPU-s | 4.23KiB/CPU-s | 0.0 | 0.035852 | False | False |
| syslog_log2metric_humio_metrics | -340.85KiB/CPU-s | -5.51 | 100.00% | 6.05MiB/CPU-s | 256.98KiB/CPU-s | 3.32KiB/CPU-s | 0.0 | 0.041505 | 5.71MiB/CPU-s | 395.96KiB/CPU-s | 5.11KiB/CPU-s | 0.0 | 0.067677 | False | False |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Regression Test Results
Run ID: 88889bcd-8b9d-440a-8d7d-8c9a595ecd75
Baseline: 4af6d78b88661d874547f336a740b845aacd9334
Comparison: 852db2e3f6ce8ff5e40a8e2cb1c2ca0046547eec
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| syslog_regex_logs2metric_ddmetrics | -187.98KiB/CPU-s | -5.15 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_datadog_logs_acks | 687.12KiB/CPU-s | 2.0 | 100.00% | 33.62MiB/CPU-s | 1.38MiB/CPU-s | 18.2KiB/CPU-s | 0.0 | 0.040953 | 34.29MiB/CPU-s | 1.05MiB/CPU-s | 13.82KiB/CPU-s | 0.0 | 0.030501 | False | False |
| syslog_splunk_hec_logs | 141.26KiB/CPU-s | 1.54 | 100.00% | 8.94MiB/CPU-s | 314.71KiB/CPU-s | 4.06KiB/CPU-s | 0.0 | 0.034363 | 9.08MiB/CPU-s | 279.35KiB/CPU-s | 3.6KiB/CPU-s | 0.0 | 0.030039 | False | False |
| splunk_hec_route_s3 | 157.39KiB/CPU-s | 1.31 | 100.00% | 11.71MiB/CPU-s | 583.25KiB/CPU-s | 7.52KiB/CPU-s | 0.0 | 0.048635 | 11.86MiB/CPU-s | 542.41KiB/CPU-s | 7.0KiB/CPU-s | 0.0 | 0.044643 | False | False |
| otlp_http_to_blackhole | 19.95KiB/CPU-s | 1.28 | 100.00% | 1.53MiB/CPU-s | 112.99KiB/CPU-s | 1.46KiB/CPU-s | 0.0 | 0.072261 | 1.55MiB/CPU-s | 117.14KiB/CPU-s | 1.51KiB/CPU-s | 0.0 | 0.073971 | False | False |
| otlp_grpc_to_blackhole | 12.68KiB/CPU-s | 1.2 | 100.00% | 1.03MiB/CPU-s | 37.62KiB/CPU-s | 497.33B/CPU-s | 0.0 | 0.035598 | 1.04MiB/CPU-s | 51.59KiB/CPU-s | 681.62B/CPU-s | 0.0 | 0.048247 | False | False |
| socket_to_socket_blackhole | 108.75KiB/CPU-s | 0.79 | 100.00% | 13.48MiB/CPU-s | 225.38KiB/CPU-s | 2.91KiB/CPU-s | 0.0 | 0.016328 | 13.58MiB/CPU-s | 270.58KiB/CPU-s | 3.49KiB/CPU-s | 0.0 | 0.01945 | False | False |
| datadog_agent_remap_blackhole_acks | 228.51KiB/CPU-s | 0.72 | 100.00% | 30.86MiB/CPU-s | 1.62MiB/CPU-s | 21.45KiB/CPU-s | 0.0 | 0.052558 | 31.08MiB/CPU-s | 1.27MiB/CPU-s | 16.84KiB/CPU-s | 0.0 | 0.040954 | False | False |
| syslog_loki | 62.25KiB/CPU-s | 0.69 | 100.00% | 8.84MiB/CPU-s | 350.12KiB/CPU-s | 4.52KiB/CPU-s | 0.0 | 0.038672 | 8.9MiB/CPU-s | 249.28KiB/CPU-s | 3.22KiB/CPU-s | 0.0 | 0.027346 | False | False |
| http_to_http_acks | 10.96KiB/CPU-s | 0.2 | 16.80% | 5.3MiB/CPU-s | 2.77MiB/CPU-s | 36.65KiB/CPU-s | 0.0 | 0.523015 | 5.31MiB/CPU-s | 2.76MiB/CPU-s | 36.42KiB/CPU-s | 0.0 | 0.518661 | True | False |
| splunk_hec_to_splunk_hec_logs_noack | 7.95KiB/CPU-s | 0.06 | 94.75% | 13.62MiB/CPU-s | 253.34KiB/CPU-s | 3.27KiB/CPU-s | 0.0 | 0.018166 | 13.63MiB/CPU-s | 191.69KiB/CPU-s | 2.47KiB/CPU-s | 0.0 | 0.013738 | False | False |
| http_to_http_noack | 8.21KiB/CPU-s | 0.06 | 79.81% | 13.61MiB/CPU-s | 391.85KiB/CPU-s | 5.05KiB/CPU-s | 0.0 | 0.028123 | 13.61MiB/CPU-s | 308.61KiB/CPU-s | 3.98KiB/CPU-s | 0.0 | 0.022136 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 1.01KiB/CPU-s | 0.01 | 13.03% | 13.62MiB/CPU-s | 340.44KiB/CPU-s | 4.39KiB/CPU-s | 0.0 | 0.024416 | 13.62MiB/CPU-s | 333.54KiB/CPU-s | 4.3KiB/CPU-s | 0.0 | 0.023919 | False | False |
| fluent_elasticsearch | -188.57B/CPU-s | -0.0 | 26.35% | 45.41MiB/CPU-s | 30.34KiB/CPU-s | 396.58B/CPU-s | 0.0 | 0.000652 | 45.41MiB/CPU-s | 30.27KiB/CPU-s | 395.81B/CPU-s | 0.0 | 0.000651 | False | False |
| splunk_hec_indexer_ack_blackhole | -1.41KiB/CPU-s | -0.01 | 25.53% | 13.62MiB/CPU-s | 232.1KiB/CPU-s | 2.99KiB/CPU-s | 0.0 | 0.016641 | 13.62MiB/CPU-s | 243.19KiB/CPU-s | 3.14KiB/CPU-s | 0.0 | 0.017438 | False | False |
| enterprise_http_to_http | -3.81KiB/CPU-s | -0.03 | 73.07% | 13.62MiB/CPU-s | 150.52KiB/CPU-s | 1.94KiB/CPU-s | 0.0 | 0.010788 | 13.62MiB/CPU-s | 220.95KiB/CPU-s | 2.85KiB/CPU-s | 0.0 | 0.015841 | False | False |
| file_to_blackhole | -14.36KiB/CPU-s | -0.03 | 49.14% | 54.49MiB/CPU-s | 1.11MiB/CPU-s | 14.61KiB/CPU-s | 0.0 | 0.020299 | 54.48MiB/CPU-s | 1.22MiB/CPU-s | 16.07KiB/CPU-s | 0.0 | 0.022349 | False | False |
| http_text_to_http_json | -13.62KiB/CPU-s | -0.05 | 70.49% | 25.38MiB/CPU-s | 723.9KiB/CPU-s | 9.35KiB/CPU-s | 0.0 | 0.027849 | 25.37MiB/CPU-s | 701.06KiB/CPU-s | 9.05KiB/CPU-s | 0.0 | 0.026985 | False | False |
| syslog_humio_logs | -30.3KiB/CPU-s | -0.32 | 100.00% | 9.12MiB/CPU-s | 283.44KiB/CPU-s | 3.66KiB/CPU-s | 0.0 | 0.030362 | 9.09MiB/CPU-s | 312.26KiB/CPU-s | 4.03KiB/CPU-s | 0.0 | 0.033559 | False | False |
| http_to_http_json | -109.14KiB/CPU-s | -0.79 | 100.00% | 13.51MiB/CPU-s | 441.01KiB/CPU-s | 5.69KiB/CPU-s | 0.0 | 0.031867 | 13.41MiB/CPU-s | 458.03KiB/CPU-s | 5.91KiB/CPU-s | 0.0 | 0.03336 | False | False |
| syslog_log2metric_splunk_hec_metrics | -85.57KiB/CPU-s | -0.88 | 100.00% | 9.52MiB/CPU-s | 127.58KiB/CPU-s | 1.65KiB/CPU-s | 0.0 | 0.013079 | 9.44MiB/CPU-s | 168.34KiB/CPU-s | 2.17KiB/CPU-s | 0.0 | 0.01741 | False | False |
| datadog_agent_remap_datadog_logs | -339.94KiB/CPU-s | -0.96 | 100.00% | 34.52MiB/CPU-s | 1.19MiB/CPU-s | 15.76KiB/CPU-s | 0.0 | 0.034555 | 34.19MiB/CPU-s | 1.46MiB/CPU-s | 19.35KiB/CPU-s | 0.0 | 0.042838 | False | False |
| datadog_agent_remap_blackhole | -733.71KiB/CPU-s | -2.28 | 100.00% | 31.36MiB/CPU-s | 353.21KiB/CPU-s | 4.56KiB/CPU-s | 0.0 | 0.010998 | 30.64MiB/CPU-s | 2.16MiB/CPU-s | 28.59KiB/CPU-s | 0.0 | 0.070564 | False | False |
| syslog_log2metric_humio_metrics | -180.16KiB/CPU-s | -2.93 | 100.00% | 6.0MiB/CPU-s | 359.69KiB/CPU-s | 4.64KiB/CPU-s | 0.0 | 0.058544 | 5.82MiB/CPU-s | 564.61KiB/CPU-s | 7.29KiB/CPU-s | 0.0 | 0.094673 | False | False |
| syslog_regex_logs2metric_ddmetrics | -187.98KiB/CPU-s | -5.15 | 100.00% | 3.57MiB/CPU-s | 510.14KiB/CPU-s | 6.58KiB/CPU-s | 0.0 | 0.139699 | 3.38MiB/CPU-s | 496.13KiB/CPU-s | 6.41KiB/CPU-s | 0.0 | 0.143238 | True | False |
Regression Test Results
Run ID: c577f6bf-4673-4510-829f-181fa5f1b638
Baseline: 69ede64bf2654256d708f3e6edf23e2b39e134a2
Comparison: 78e09cfcae6ed1c06b260889fb31dc50793e8b24
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| http_text_to_http_json | 1.36MiB/CPU-s | 5.68 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_text_to_http_json | 1.36MiB/CPU-s | 5.68 | 100.00% | 23.92MiB/CPU-s | 628.05KiB/CPU-s | 8.11KiB/CPU-s | 0.0 | 0.025641 | 25.28MiB/CPU-s | 797.9KiB/CPU-s | 10.29KiB/CPU-s | 0.0 | 0.030826 | False | False |
| syslog_humio_logs | 233.42KiB/CPU-s | 2.54 | 100.00% | 8.98MiB/CPU-s | 373.98KiB/CPU-s | 4.83KiB/CPU-s | 0.0 | 0.040689 | 9.2MiB/CPU-s | 101.75KiB/CPU-s | 1.31KiB/CPU-s | 0.0 | 0.010796 | False | False |
| datadog_agent_remap_datadog_logs | 829.14KiB/CPU-s | 2.4 | 100.00% | 33.67MiB/CPU-s | 1.53MiB/CPU-s | 20.15KiB/CPU-s | 0.0 | 0.045305 | 34.48MiB/CPU-s | 1.34MiB/CPU-s | 17.67KiB/CPU-s | 0.0 | 0.03879 | False | False |
| socket_to_socket_blackhole | 278.01KiB/CPU-s | 2.03 | 100.00% | 13.35MiB/CPU-s | 345.14KiB/CPU-s | 4.45KiB/CPU-s | 0.0 | 0.025253 | 13.62MiB/CPU-s | 285.67KiB/CPU-s | 3.69KiB/CPU-s | 0.0 | 0.020485 | False | False |
| datadog_agent_remap_blackhole | 602.94KiB/CPU-s | 1.9 | 100.00% | 31.03MiB/CPU-s | 1.41MiB/CPU-s | 18.62KiB/CPU-s | 0.0 | 0.045355 | 31.62MiB/CPU-s | 376.58KiB/CPU-s | 4.86KiB/CPU-s | 0.0 | 0.011629 | False | False |
| http_to_http_acks | 98.64KiB/CPU-s | 1.85 | 94.49% | 5.2MiB/CPU-s | 2.71MiB/CPU-s | 35.87KiB/CPU-s | 0.0 | 0.522042 | 5.3MiB/CPU-s | 2.79MiB/CPU-s | 36.84KiB/CPU-s | 0.0 | 0.526381 | True | False |
| splunk_hec_route_s3 | 137.55KiB/CPU-s | 1.15 | 100.00% | 11.71MiB/CPU-s | 559.37KiB/CPU-s | 7.22KiB/CPU-s | 0.0 | 0.046657 | 11.84MiB/CPU-s | 517.76KiB/CPU-s | 6.68KiB/CPU-s | 0.0 | 0.042696 | False | False |
| syslog_splunk_hec_logs | 101.78KiB/CPU-s | 1.1 | 100.00% | 9.0MiB/CPU-s | 213.31KiB/CPU-s | 2.75KiB/CPU-s | 0.0 | 0.023139 | 9.1MiB/CPU-s | 199.95KiB/CPU-s | 2.58KiB/CPU-s | 0.0 | 0.021454 | False | False |
| otlp_http_to_blackhole | 16.64KiB/CPU-s | 1.06 | 100.00% | 1.53MiB/CPU-s | 119.87KiB/CPU-s | 1.55KiB/CPU-s | 0.0 | 0.076394 | 1.55MiB/CPU-s | 109.64KiB/CPU-s | 1.41KiB/CPU-s | 0.0 | 0.069144 | False | False |
| syslog_loki | 30.11KiB/CPU-s | 0.33 | 100.00% | 8.82MiB/CPU-s | 252.49KiB/CPU-s | 3.26KiB/CPU-s | 0.0 | 0.027958 | 8.85MiB/CPU-s | 262.88KiB/CPU-s | 3.39KiB/CPU-s | 0.0 | 0.029011 | False | False |
| datadog_agent_remap_datadog_logs_acks | 44.87KiB/CPU-s | 0.13 | 96.01% | 34.03MiB/CPU-s | 928.87KiB/CPU-s | 11.99KiB/CPU-s | 0.0 | 0.026655 | 34.07MiB/CPU-s | 1.38MiB/CPU-s | 18.25KiB/CPU-s | 0.0 | 0.040523 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 6.04KiB/CPU-s | 0.04 | 87.67% | 13.62MiB/CPU-s | 235.53KiB/CPU-s | 3.04KiB/CPU-s | 0.0 | 0.016887 | 13.63MiB/CPU-s | 191.51KiB/CPU-s | 2.47KiB/CPU-s | 0.0 | 0.013725 | False | False |
| http_to_http_noack | 2.08KiB/CPU-s | 0.01 | 24.70% | 13.61MiB/CPU-s | 372.23KiB/CPU-s | 4.8KiB/CPU-s | 0.0 | 0.026711 | 13.61MiB/CPU-s | 351.52KiB/CPU-s | 4.54KiB/CPU-s | 0.0 | 0.025221 | False | False |
| enterprise_http_to_http | 2.01KiB/CPU-s | 0.01 | 39.97% | 13.62MiB/CPU-s | 228.64KiB/CPU-s | 2.95KiB/CPU-s | 0.0 | 0.016393 | 13.62MiB/CPU-s | 190.5KiB/CPU-s | 2.46KiB/CPU-s | 0.0 | 0.013656 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 1.22KiB/CPU-s | 0.01 | 15.11% | 13.61MiB/CPU-s | 354.38KiB/CPU-s | 4.57KiB/CPU-s | 0.0 | 0.02542 | 13.61MiB/CPU-s | 346.87KiB/CPU-s | 4.47KiB/CPU-s | 0.0 | 0.02488 | False | False |
| fluent_elasticsearch | -163.66B/CPU-s | -0.0 | 23.18% | 45.41MiB/CPU-s | 30.1KiB/CPU-s | 393.52B/CPU-s | 0.0 | 0.000647 | 45.41MiB/CPU-s | 29.97KiB/CPU-s | 391.75B/CPU-s | 0.0 | 0.000644 | False | False |
| file_to_blackhole | -8.18KiB/CPU-s | -0.01 | 30.19% | 54.5MiB/CPU-s | 1.16MiB/CPU-s | 15.3KiB/CPU-s | 0.0 | 0.02125 | 54.49MiB/CPU-s | 1.1MiB/CPU-s | 14.52KiB/CPU-s | 0.0 | 0.020171 | False | False |
| splunk_hec_indexer_ack_blackhole | -1.92KiB/CPU-s | -0.01 | 32.60% | 13.62MiB/CPU-s | 246.04KiB/CPU-s | 3.17KiB/CPU-s | 0.0 | 0.017643 | 13.62MiB/CPU-s | 255.5KiB/CPU-s | 3.3KiB/CPU-s | 0.0 | 0.018324 | False | False |
| http_to_http_json | -8.58KiB/CPU-s | -0.06 | 94.31% | 13.6MiB/CPU-s | 242.54KiB/CPU-s | 3.13KiB/CPU-s | 0.0 | 0.01741 | 13.59MiB/CPU-s | 251.15KiB/CPU-s | 3.24KiB/CPU-s | 0.0 | 0.01804 | False | False |
| syslog_log2metric_humio_metrics | -4.17KiB/CPU-s | -0.07 | 86.93% | 6.17MiB/CPU-s | 112.11KiB/CPU-s | 1.45KiB/CPU-s | 0.0 | 0.017746 | 6.16MiB/CPU-s | 181.7KiB/CPU-s | 2.35KiB/CPU-s | 0.0 | 0.02878 | False | False |
| syslog_log2metric_splunk_hec_metrics | -14.72KiB/CPU-s | -0.16 | 98.35% | 9.17MiB/CPU-s | 337.54KiB/CPU-s | 4.36KiB/CPU-s | 0.0 | 0.035942 | 9.16MiB/CPU-s | 334.47KiB/CPU-s | 4.32KiB/CPU-s | 0.0 | 0.035672 | False | False |
| otlp_grpc_to_blackhole | -13.94KiB/CPU-s | -1.3 | 100.00% | 1.05MiB/CPU-s | 34.83KiB/CPU-s | 460.39B/CPU-s | 0.0 | 0.032532 | 1.03MiB/CPU-s | 55.59KiB/CPU-s | 734.54B/CPU-s | 0.0 | 0.052615 | False | False |
| datadog_agent_remap_blackhole_acks | -582.78KiB/CPU-s | -1.78 | 100.00% | 31.97MiB/CPU-s | 614.11KiB/CPU-s | 7.93KiB/CPU-s | 0.0 | 0.01876 | 31.4MiB/CPU-s | 1.15MiB/CPU-s | 15.23KiB/CPU-s | 0.0 | 0.036677 | False | False |
| syslog_regex_logs2metric_ddmetrics | -72.86KiB/CPU-s | -1.89 | 100.00% | 3.76MiB/CPU-s | 439.14KiB/CPU-s | 5.67KiB/CPU-s | 0.0 | 0.114146 | 3.69MiB/CPU-s | 478.89KiB/CPU-s | 6.18KiB/CPU-s | 0.0 | 0.126882 | True | False |
Regression Test Results
Run ID: 20fe7d4d-ab71-4328-a047-41ef7abc695a
Baseline: 69ede64bf2654256d708f3e6edf23e2b39e134a2
Comparison: 6776823a44cf76702530eed6630cf7f3744944a2
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| http_text_to_http_json | 1.55MiB/CPU-s | 6.52 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_text_to_http_json | 1.55MiB/CPU-s | 6.52 | 100.00% | 23.74MiB/CPU-s | 664.05KiB/CPU-s | 8.57KiB/CPU-s | 0.0 | 0.027318 | 25.28MiB/CPU-s | 583.09KiB/CPU-s | 7.52KiB/CPU-s | 0.0 | 0.022519 | False | False |
| datadog_agent_remap_datadog_logs | 1.12MiB/CPU-s | 3.36 | 100.00% | 33.35MiB/CPU-s | 1.52MiB/CPU-s | 20.13KiB/CPU-s | 0.0 | 0.045682 | 34.48MiB/CPU-s | 1.19MiB/CPU-s | 15.71KiB/CPU-s | 0.0 | 0.034501 | False | False |
| datadog_agent_remap_datadog_logs_acks | 862.13KiB/CPU-s | 2.59 | 100.00% | 32.56MiB/CPU-s | 1.72MiB/CPU-s | 22.77KiB/CPU-s | 0.0 | 0.052925 | 33.41MiB/CPU-s | 1.76MiB/CPU-s | 23.32KiB/CPU-s | 0.0 | 0.05282 | False | False |
| syslog_log2metric_splunk_hec_metrics | 197.7KiB/CPU-s | 2.07 | 100.00% | 9.32MiB/CPU-s | 445.14KiB/CPU-s | 5.75KiB/CPU-s | 0.0 | 0.046658 | 9.51MiB/CPU-s | 94.28KiB/CPU-s | 1.22KiB/CPU-s | 0.0 | 0.009681 | False | False |
| otlp_http_to_blackhole | 7.18KiB/CPU-s | 0.46 | 99.84% | 1.51MiB/CPU-s | 123.37KiB/CPU-s | 1.59KiB/CPU-s | 0.0 | 0.079844 | 1.52MiB/CPU-s | 125.51KiB/CPU-s | 1.62KiB/CPU-s | 0.0 | 0.080854 | False | False |
| otlp_grpc_to_blackhole | 3.37KiB/CPU-s | 0.32 | 99.99% | 1.04MiB/CPU-s | 43.8KiB/CPU-s | 578.87B/CPU-s | 0.0 | 0.041138 | 1.04MiB/CPU-s | 48.41KiB/CPU-s | 639.59B/CPU-s | 0.0 | 0.045321 | False | False |
| enterprise_http_to_http | 14.46KiB/CPU-s | 0.1 | 99.21% | 13.61MiB/CPU-s | 375.71KiB/CPU-s | 4.85KiB/CPU-s | 0.0 | 0.026961 | 13.62MiB/CPU-s | 191.56KiB/CPU-s | 2.47KiB/CPU-s | 0.0 | 0.013732 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 6.99KiB/CPU-s | 0.05 | 87.31% | 13.62MiB/CPU-s | 298.85KiB/CPU-s | 3.86KiB/CPU-s | 0.0 | 0.021428 | 13.63MiB/CPU-s | 190.82KiB/CPU-s | 2.46KiB/CPU-s | 0.0 | 0.013675 | False | False |
| fluent_elasticsearch | 11.53B/CPU-s | 0.0 | 1.65% | 45.41MiB/CPU-s | 30.41KiB/CPU-s | 397.62B/CPU-s | 0.0 | 0.000654 | 45.41MiB/CPU-s | 29.79KiB/CPU-s | 389.45B/CPU-s | 0.0 | 0.000641 | False | False |
| http_to_http_noack | -111.36B/CPU-s | -0.0 | 1.54% | 13.61MiB/CPU-s | 305.73KiB/CPU-s | 3.95KiB/CPU-s | 0.0 | 0.021929 | 13.61MiB/CPU-s | 309.8KiB/CPU-s | 4.0KiB/CPU-s | 0.0 | 0.022221 | False | False |
| splunk_hec_indexer_ack_blackhole | 332.71B/CPU-s | 0.0 | 5.57% | 13.62MiB/CPU-s | 255.6KiB/CPU-s | 3.3KiB/CPU-s | 0.0 | 0.018331 | 13.62MiB/CPU-s | 254.37KiB/CPU-s | 3.28KiB/CPU-s | 0.0 | 0.018242 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | -1.31KiB/CPU-s | -0.01 | 16.32% | 13.62MiB/CPU-s | 350.67KiB/CPU-s | 4.52KiB/CPU-s | 0.0 | 0.025147 | 13.62MiB/CPU-s | 346.49KiB/CPU-s | 4.47KiB/CPU-s | 0.0 | 0.02485 | False | False |
| file_to_blackhole | -7.36KiB/CPU-s | -0.01 | 27.96% | 54.5MiB/CPU-s | 1.08MiB/CPU-s | 14.31KiB/CPU-s | 0.0 | 0.01988 | 54.49MiB/CPU-s | 1.12MiB/CPU-s | 14.75KiB/CPU-s | 0.0 | 0.020507 | False | False |
| syslog_log2metric_humio_metrics | -6.97KiB/CPU-s | -0.11 | 97.74% | 6.17MiB/CPU-s | 103.78KiB/CPU-s | 1.34KiB/CPU-s | 0.0 | 0.016435 | 6.16MiB/CPU-s | 212.82KiB/CPU-s | 2.75KiB/CPU-s | 0.0 | 0.033739 | False | False |
| splunk_hec_route_s3 | -20.04KiB/CPU-s | -0.17 | 86.91% | 11.6MiB/CPU-s | 663.1KiB/CPU-s | 8.55KiB/CPU-s | 0.0 | 0.055834 | 11.58MiB/CPU-s | 785.4KiB/CPU-s | 10.14KiB/CPU-s | 0.0 | 0.066244 | False | False |
| http_to_http_json | -28.35KiB/CPU-s | -0.2 | 100.00% | 13.58MiB/CPU-s | 286.08KiB/CPU-s | 3.69KiB/CPU-s | 0.0 | 0.020577 | 13.55MiB/CPU-s | 334.45KiB/CPU-s | 4.31KiB/CPU-s | 0.0 | 0.024105 | False | False |
| syslog_splunk_hec_logs | -77.55KiB/CPU-s | -0.85 | 100.00% | 8.89MiB/CPU-s | 183.64KiB/CPU-s | 2.37KiB/CPU-s | 0.0 | 0.020165 | 8.82MiB/CPU-s | 303.75KiB/CPU-s | 3.92KiB/CPU-s | 0.0 | 0.03364 | False | False |
| http_to_http_acks | -54.27KiB/CPU-s | -1.0 | 70.08% | 5.29MiB/CPU-s | 2.81MiB/CPU-s | 37.12KiB/CPU-s | 0.0 | 0.530832 | 5.24MiB/CPU-s | 2.78MiB/CPU-s | 36.79KiB/CPU-s | 0.0 | 0.531403 | True | False |
| datadog_agent_remap_blackhole_acks | -375.06KiB/CPU-s | -1.16 | 100.00% | 31.53MiB/CPU-s | 790.37KiB/CPU-s | 10.21KiB/CPU-s | 0.0 | 0.024477 | 31.16MiB/CPU-s | 1.49MiB/CPU-s | 19.66KiB/CPU-s | 0.0 | 0.047718 | False | False |
| syslog_humio_logs | -111.69KiB/CPU-s | -1.18 | 100.00% | 9.21MiB/CPU-s | 156.47KiB/CPU-s | 2.02KiB/CPU-s | 0.0 | 0.016589 | 9.1MiB/CPU-s | 123.6KiB/CPU-s | 1.6KiB/CPU-s | 0.0 | 0.013262 | False | False |
| socket_to_socket_blackhole | -280.08KiB/CPU-s | -2.04 | 100.00% | 13.38MiB/CPU-s | 275.33KiB/CPU-s | 3.55KiB/CPU-s | 0.0 | 0.020092 | 13.11MiB/CPU-s | 850.93KiB/CPU-s | 10.99KiB/CPU-s | 0.0 | 0.063391 | False | False |
| syslog_loki | -210.08KiB/CPU-s | -2.32 | 100.00% | 8.83MiB/CPU-s | 182.97KiB/CPU-s | 2.36KiB/CPU-s | 0.0 | 0.020225 | 8.63MiB/CPU-s | 435.7KiB/CPU-s | 5.62KiB/CPU-s | 0.0 | 0.049305 | False | False |
| datadog_agent_remap_blackhole | -1.29MiB/CPU-s | -4.21 | 100.00% | 30.67MiB/CPU-s | 1.64MiB/CPU-s | 21.74KiB/CPU-s | 0.0 | 0.053593 | 29.38MiB/CPU-s | 1.73MiB/CPU-s | 22.83KiB/CPU-s | 0.0 | 0.058789 | False | False |
| syslog_regex_logs2metric_ddmetrics | -181.04KiB/CPU-s | -4.8 | 100.00% | 3.68MiB/CPU-s | 511.7KiB/CPU-s | 6.61KiB/CPU-s | 0.0 | 0.135644 | 3.51MiB/CPU-s | 521.39KiB/CPU-s | 6.73KiB/CPU-s | 0.0 | 0.145178 | True | False |
This change also drops https://vector.dev/docs/reference/configuration/sources/gcp_pubsub/#gcp-pub/sub and https://vector.dev/docs/reference/configuration/sources/gcp_pubsub/#proxy
This change also drops https://vector.dev/docs/reference/configuration/sources/gcp_pubsub/#gcp-pub/sub and https://vector.dev/docs/reference/configuration/sources/gcp_pubsub/#proxy
Good catch
Regression Test Results
Run ID: df04f595-cb6d-4747-8a1d-765eab4fd4f9
Baseline: f90ca9e3e82b7577608b0c82a5feffb3f96d5f4c
Comparison: 90dfcf913b556ab3cbf0cc4e515472018bac2a99
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| http_text_to_http_json | 1.37MiB/CPU-s | 5.69 | 100.00% |
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_text_to_http_json | 1.37MiB/CPU-s | 5.69 | 100.00% | 24.02MiB/CPU-s | 1.15MiB/CPU-s | 15.16KiB/CPU-s | 0.0 | 0.047732 | 25.39MiB/CPU-s | 561.01KiB/CPU-s | 7.24KiB/CPU-s | 0.0 | 0.021577 | False | False |
| splunk_hec_route_s3 | 287.99KiB/CPU-s | 2.42 | 100.00% | 11.62MiB/CPU-s | 599.16KiB/CPU-s | 7.73KiB/CPU-s | 0.0 | 0.050348 | 11.9MiB/CPU-s | 535.65KiB/CPU-s | 6.91KiB/CPU-s | 0.0 | 0.043948 | False | False |
| socket_to_socket_blackhole | 170.11KiB/CPU-s | 1.25 | 100.00% | 13.32MiB/CPU-s | 295.18KiB/CPU-s | 3.81KiB/CPU-s | 0.0 | 0.021636 | 13.49MiB/CPU-s | 183.28KiB/CPU-s | 2.37KiB/CPU-s | 0.0 | 0.013268 | False | False |
| datadog_agent_remap_blackhole | 198.85KiB/CPU-s | 0.63 | 100.00% | 30.93MiB/CPU-s | 1.7MiB/CPU-s | 22.49KiB/CPU-s | 0.0 | 0.055012 | 31.13MiB/CPU-s | 1.33MiB/CPU-s | 17.58KiB/CPU-s | 0.0 | 0.042743 | False | False |
| syslog_loki | 34.37KiB/CPU-s | 0.38 | 100.00% | 8.76MiB/CPU-s | 329.26KiB/CPU-s | 4.25KiB/CPU-s | 0.0 | 0.036713 | 8.79MiB/CPU-s | 252.29KiB/CPU-s | 3.26KiB/CPU-s | 0.0 | 0.028023 | False | False |
| http_to_http_json | 49.19KiB/CPU-s | 0.35 | 100.00% | 13.57MiB/CPU-s | 286.48KiB/CPU-s | 3.7KiB/CPU-s | 0.0 | 0.020608 | 13.62MiB/CPU-s | 212.05KiB/CPU-s | 2.74KiB/CPU-s | 0.0 | 0.0152 | False | False |
| datadog_agent_remap_datadog_logs | 39.32KiB/CPU-s | 0.11 | 91.02% | 33.6MiB/CPU-s | 1.17MiB/CPU-s | 15.43KiB/CPU-s | 0.0 | 0.034763 | 33.64MiB/CPU-s | 1.31MiB/CPU-s | 17.3KiB/CPU-s | 0.0 | 0.038927 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 7.71KiB/CPU-s | 0.06 | 94.64% | 13.62MiB/CPU-s | 244.05KiB/CPU-s | 3.15KiB/CPU-s | 0.0 | 0.0175 | 13.63MiB/CPU-s | 190.05KiB/CPU-s | 2.45KiB/CPU-s | 0.0 | 0.01362 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 3.1KiB/CPU-s | 0.02 | 37.15% | 13.61MiB/CPU-s | 367.07KiB/CPU-s | 4.73KiB/CPU-s | 0.0 | 0.026329 | 13.62MiB/CPU-s | 333.8KiB/CPU-s | 4.31KiB/CPU-s | 0.0 | 0.023937 | False | False |
| file_to_blackhole | 3.26KiB/CPU-s | 0.01 | 12.28% | 54.49MiB/CPU-s | 1.14MiB/CPU-s | 15.07KiB/CPU-s | 0.0 | 0.020933 | 54.49MiB/CPU-s | 1.12MiB/CPU-s | 14.72KiB/CPU-s | 0.0 | 0.020465 | False | False |
| enterprise_http_to_http | 6.36B/CPU-s | 0.0 | 0.14% | 13.62MiB/CPU-s | 187.08KiB/CPU-s | 2.42KiB/CPU-s | 0.0 | 0.013411 | 13.62MiB/CPU-s | 192.08KiB/CPU-s | 2.48KiB/CPU-s | 0.0 | 0.013769 | False | False |
| splunk_hec_indexer_ack_blackhole | 394.73B/CPU-s | 0.0 | 6.68% | 13.62MiB/CPU-s | 254.0KiB/CPU-s | 3.28KiB/CPU-s | 0.0 | 0.018215 | 13.62MiB/CPU-s | 250.3KiB/CPU-s | 3.23KiB/CPU-s | 0.0 | 0.017949 | False | False |
| fluent_elasticsearch | -3.82KiB/CPU-s | -0.01 | 68.14% | 45.41MiB/CPU-s | 29.93KiB/CPU-s | 391.25B/CPU-s | 0.0 | 0.000643 | 45.41MiB/CPU-s | 298.62KiB/CPU-s | 3.81KiB/CPU-s | 0.0 | 0.006422 | False | False |
| http_to_http_noack | -4.19KiB/CPU-s | -0.03 | 50.85% | 13.61MiB/CPU-s | 310.51KiB/CPU-s | 4.01KiB/CPU-s | 0.0 | 0.022272 | 13.61MiB/CPU-s | 355.34KiB/CPU-s | 4.58KiB/CPU-s | 0.0 | 0.025495 | False | False |
| syslog_log2metric_humio_metrics | -12.94KiB/CPU-s | -0.22 | 90.88% | 5.86MiB/CPU-s | 424.4KiB/CPU-s | 5.48KiB/CPU-s | 0.0 | 0.070769 | 5.84MiB/CPU-s | 414.85KiB/CPU-s | 5.35KiB/CPU-s | 0.0 | 0.069326 | False | False |
| datadog_agent_remap_datadog_logs_acks | -223.45KiB/CPU-s | -0.64 | 100.00% | 34.11MiB/CPU-s | 905.65KiB/CPU-s | 11.69KiB/CPU-s | 0.0 | 0.025926 | 33.89MiB/CPU-s | 1.17MiB/CPU-s | 15.43KiB/CPU-s | 0.0 | 0.034435 | False | False |
| otlp_grpc_to_blackhole | -9.08KiB/CPU-s | -0.85 | 100.00% | 1.04MiB/CPU-s | 38.44KiB/CPU-s | 508.19B/CPU-s | 0.0 | 0.036088 | 1.03MiB/CPU-s | 51.32KiB/CPU-s | 678.13B/CPU-s | 0.0 | 0.048593 | False | False |
| syslog_regex_logs2metric_ddmetrics | -35.76KiB/CPU-s | -0.97 | 100.00% | 3.6MiB/CPU-s | 468.05KiB/CPU-s | 6.04KiB/CPU-s | 0.0 | 0.127005 | 3.56MiB/CPU-s | 438.41KiB/CPU-s | 5.66KiB/CPU-s | 0.0 | 0.120129 | True | False |
| syslog_log2metric_splunk_hec_metrics | -208.11KiB/CPU-s | -2.15 | 100.00% | 9.46MiB/CPU-s | 373.81KiB/CPU-s | 4.83KiB/CPU-s | 0.0 | 0.038577 | 9.26MiB/CPU-s | 462.98KiB/CPU-s | 5.98KiB/CPU-s | 0.0 | 0.048827 | False | False |
| otlp_http_to_blackhole | -34.52KiB/CPU-s | -2.17 | 100.00% | 1.55MiB/CPU-s | 108.0KiB/CPU-s | 1.39KiB/CPU-s | 0.0 | 0.068046 | 1.52MiB/CPU-s | 127.24KiB/CPU-s | 1.64KiB/CPU-s | 0.0 | 0.08195 | False | False |
| http_to_http_acks | -128.46KiB/CPU-s | -2.34 | 98.65% | 5.36MiB/CPU-s | 2.78MiB/CPU-s | 36.69KiB/CPU-s | 0.0 | 0.517969 | 5.23MiB/CPU-s | 2.79MiB/CPU-s | 36.82KiB/CPU-s | 0.0 | 0.532296 | True | False |
| syslog_humio_logs | -278.95KiB/CPU-s | -2.97 | 100.00% | 9.17MiB/CPU-s | 297.94KiB/CPU-s | 3.85KiB/CPU-s | 0.0 | 0.031742 | 8.89MiB/CPU-s | 253.03KiB/CPU-s | 3.27KiB/CPU-s | 0.0 | 0.027784 | False | False |
| syslog_splunk_hec_logs | -351.61KiB/CPU-s | -3.8 | 100.00% | 9.04MiB/CPU-s | 230.83KiB/CPU-s | 2.98KiB/CPU-s | 0.0 | 0.024926 | 8.7MiB/CPU-s | 273.98KiB/CPU-s | 3.53KiB/CPU-s | 0.0 | 0.030753 | False | False |
| datadog_agent_remap_blackhole_acks | -1.41MiB/CPU-s | -4.44 | 100.00% | 31.74MiB/CPU-s | 1.33MiB/CPU-s | 17.58KiB/CPU-s | 0.0 | 0.041884 | 30.33MiB/CPU-s | 1.74MiB/CPU-s | 22.97KiB/CPU-s | 0.0 | 0.057286 | False | False |
Regression Test Results
Run ID: fba49258-5e6a-49a7-bf76-9652e9088f64
Baseline: ce003848558b7ee602336c7f2f0feda9e675e0ab
Comparison: e3b480d0be52f429d26051eb82e1f199c07e8783
Total vector CPUs: 7
Explanation
A regression test is an integrated performance test for vector in
a repeatable rig, with varying configuration for vector. What
follows is a statistical summary of a brief vector run for each
configuration across SHAs given above. The goal of these tests are to determine,
quickly, if vector performance is changed and to what degree by a
pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a
statistically significant change in their bytes_written_per_cpu_second performance
between baseline and comparison SHAs, with 90.0%
confidence OR have been detected as newly erratic. Negative values mean that
baseline is faster, positive comparison. Results that do not exhibit more than a
±5% change in mean bytes_written_per_cpu_second are
discarded. An experiment is erratic if its coefficient of variation is greater
than 0.1. The abbreviated table will be
omitted if no interesting changes are observed.
No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| socket_to_socket_blackhole | 310.02KiB/CPU-s | 2.28 | 100.00% | 13.29MiB/CPU-s | 197.83KiB/CPU-s | 2.55KiB/CPU-s | 0.0 | 0.014534 | 13.59MiB/CPU-s | 136.83KiB/CPU-s | 1.77KiB/CPU-s | 0.0 | 0.009829 | False | False |
| http_text_to_http_json | 397.97KiB/CPU-s | 1.57 | 100.00% | 24.81MiB/CPU-s | 581.71KiB/CPU-s | 7.51KiB/CPU-s | 0.0 | 0.022892 | 25.2MiB/CPU-s | 731.19KiB/CPU-s | 9.43KiB/CPU-s | 0.0 | 0.028331 | False | False |
| http_to_http_json | 83.28KiB/CPU-s | 0.6 | 100.00% | 13.54MiB/CPU-s | 349.16KiB/CPU-s | 4.5KiB/CPU-s | 0.0 | 0.025179 | 13.62MiB/CPU-s | 208.92KiB/CPU-s | 2.7KiB/CPU-s | 0.0 | 0.014976 | False | False |
| otlp_grpc_to_blackhole | 3.56KiB/CPU-s | 0.34 | 99.97% | 1.03MiB/CPU-s | 56.57KiB/CPU-s | 747.61B/CPU-s | 0.0 | 0.05376 | 1.03MiB/CPU-s | 50.42KiB/CPU-s | 666.27B/CPU-s | 0.0 | 0.047754 | False | False |
| enterprise_http_to_http | 16.52KiB/CPU-s | 0.12 | 99.86% | 13.61MiB/CPU-s | 371.24KiB/CPU-s | 4.79KiB/CPU-s | 0.0 | 0.02664 | 13.62MiB/CPU-s | 149.01KiB/CPU-s | 1.92KiB/CPU-s | 0.0 | 0.01068 | False | False |
| syslog_log2metric_humio_metrics | 7.12KiB/CPU-s | 0.11 | 88.26% | 6.1MiB/CPU-s | 233.51KiB/CPU-s | 3.01KiB/CPU-s | 0.0 | 0.037353 | 6.11MiB/CPU-s | 263.34KiB/CPU-s | 3.4KiB/CPU-s | 0.0 | 0.042076 | False | False |
| http_to_http_noack | 8.26KiB/CPU-s | 0.06 | 88.83% | 13.61MiB/CPU-s | 332.16KiB/CPU-s | 4.29KiB/CPU-s | 0.0 | 0.023828 | 13.62MiB/CPU-s | 227.26KiB/CPU-s | 2.93KiB/CPU-s | 0.0 | 0.016293 | False | False |
| splunk_hec_to_splunk_hec_logs_noack | 8.91KiB/CPU-s | 0.06 | 96.90% | 13.62MiB/CPU-s | 257.21KiB/CPU-s | 3.32KiB/CPU-s | 0.0 | 0.018445 | 13.63MiB/CPU-s | 190.32KiB/CPU-s | 2.46KiB/CPU-s | 0.0 | 0.01364 | False | False |
| splunk_hec_to_splunk_hec_logs_acks | 1.26KiB/CPU-s | 0.01 | 15.73% | 13.62MiB/CPU-s | 341.13KiB/CPU-s | 4.4KiB/CPU-s | 0.0 | 0.024463 | 13.62MiB/CPU-s | 351.86KiB/CPU-s | 4.54KiB/CPU-s | 0.0 | 0.02523 | False | False |
| fluent_elasticsearch | -216.48B/CPU-s | -0.0 | 30.27% | 45.41MiB/CPU-s | 30.29KiB/CPU-s | 395.74B/CPU-s | 0.0 | 0.000651 | 45.41MiB/CPU-s | 29.94KiB/CPU-s | 391.42B/CPU-s | 0.0 | 0.000644 | False | False |
| splunk_hec_indexer_ack_blackhole | -1.53KiB/CPU-s | -0.01 | 25.83% | 13.62MiB/CPU-s | 251.9KiB/CPU-s | 3.25KiB/CPU-s | 0.0 | 0.018064 | 13.62MiB/CPU-s | 255.55KiB/CPU-s | 3.3KiB/CPU-s | 0.0 | 0.018327 | False | False |
| file_to_blackhole | -15.92KiB/CPU-s | -0.03 | 52.34% | 54.49MiB/CPU-s | 1.17MiB/CPU-s | 15.43KiB/CPU-s | 0.0 | 0.021434 | 54.48MiB/CPU-s | 1.23MiB/CPU-s | 16.19KiB/CPU-s | 0.0 | 0.022521 | False | False |
| splunk_hec_route_s3 | -6.24KiB/CPU-s | -0.05 | 46.58% | 11.73MiB/CPU-s | 564.15KiB/CPU-s | 7.28KiB/CPU-s | 0.0 | 0.046956 | 11.73MiB/CPU-s | 536.64KiB/CPU-s | 6.93KiB/CPU-s | 0.0 | 0.044689 | False | False |
| datadog_agent_remap_blackhole_acks | -140.26KiB/CPU-s | -0.43 | 100.00% | 31.57MiB/CPU-s | 1013.4KiB/CPU-s | 13.08KiB/CPU-s | 0.0 | 0.031341 | 31.44MiB/CPU-s | 1.18MiB/CPU-s | 15.63KiB/CPU-s | 0.0 | 0.037613 | False | False |
| syslog_loki | -43.79KiB/CPU-s | -0.49 | 100.00% | 8.71MiB/CPU-s | 246.66KiB/CPU-s | 3.18KiB/CPU-s | 0.0 | 0.027644 | 8.67MiB/CPU-s | 357.43KiB/CPU-s | 4.61KiB/CPU-s | 0.0 | 0.040256 | False | False |
| datadog_agent_remap_blackhole | -175.46KiB/CPU-s | -0.55 | 100.00% | 30.93MiB/CPU-s | 881.46KiB/CPU-s | 11.38KiB/CPU-s | 0.0 | 0.027829 | 30.76MiB/CPU-s | 1021.53KiB/CPU-s | 13.19KiB/CPU-s | 0.0 | 0.032431 | False | False |
| datadog_agent_remap_datadog_logs_acks | -205.16KiB/CPU-s | -0.59 | 100.00% | 34.15MiB/CPU-s | 1.06MiB/CPU-s | 14.0KiB/CPU-s | 0.0 | 0.031026 | 33.95MiB/CPU-s | 1.07MiB/CPU-s | 14.14KiB/CPU-s | 0.0 | 0.031527 | False | False |
| syslog_log2metric_splunk_hec_metrics | -83.21KiB/CPU-s | -0.87 | 100.00% | 9.34MiB/CPU-s | 193.95KiB/CPU-s | 2.5KiB/CPU-s | 0.0 | 0.020286 | 9.26MiB/CPU-s | 311.12KiB/CPU-s | 4.02KiB/CPU-s | 0.0 | 0.032826 | False | False |
| datadog_agent_remap_datadog_logs | -311.18KiB/CPU-s | -0.88 | 100.00% | 34.67MiB/CPU-s | 1.15MiB/CPU-s | 15.24KiB/CPU-s | 0.0 | 0.033288 | 34.36MiB/CPU-s | 1.17MiB/CPU-s | 15.5KiB/CPU-s | 0.0 | 0.034151 | False | False |
| http_to_http_acks | -55.41KiB/CPU-s | -1.02 | 71.33% | 5.29MiB/CPU-s | 2.76MiB/CPU-s | 36.43KiB/CPU-s | 0.0 | 0.521404 | 5.23MiB/CPU-s | 2.81MiB/CPU-s | 37.11KiB/CPU-s | 0.0 | 0.536677 | True | False |
| syslog_regex_logs2metric_ddmetrics | -50.86KiB/CPU-s | -1.35 | 100.00% | 3.67MiB/CPU-s | 390.31KiB/CPU-s | 5.04KiB/CPU-s | 0.0 | 0.103731 | 3.62MiB/CPU-s | 410.67KiB/CPU-s | 5.3KiB/CPU-s | 0.0 | 0.110639 | True | False |
| syslog_splunk_hec_logs | -126.3KiB/CPU-s | -1.36 | 100.00% | 9.05MiB/CPU-s | 141.65KiB/CPU-s | 1.83KiB/CPU-s | 0.0 | 0.015287 | 8.93MiB/CPU-s | 194.75KiB/CPU-s | 2.51KiB/CPU-s | 0.0 | 0.021307 | False | False |
| syslog_humio_logs | -154.96KiB/CPU-s | -1.66 | 100.00% | 9.1MiB/CPU-s | 141.87KiB/CPU-s | 1.83KiB/CPU-s | 0.0 | 0.015215 | 8.95MiB/CPU-s | 210.66KiB/CPU-s | 2.72KiB/CPU-s | 0.0 | 0.022975 | False | False |
| otlp_http_to_blackhole | -29.56KiB/CPU-s | -1.89 | 100.00% | 1.53MiB/CPU-s | 110.68KiB/CPU-s | 1.43KiB/CPU-s | 0.0 | 0.070706 | 1.5MiB/CPU-s | 132.53KiB/CPU-s | 1.71KiB/CPU-s | 0.0 | 0.086296 | False | False |