vector icon indicating copy to clipboard operation
vector copied to clipboard

docs(dnstap source): Use generated docs for configuration

Open StephenWakely opened this issue 2 years ago • 6 comments

Closes #15830 Epic #14998

~~Keeping this in draft for now because there seem to be issues with setting the units for optional fields. Also I'm not sure if I have added the warnings properly.~~

StephenWakely avatar Jan 11 '23 13:01 StephenWakely

Deploy Preview for vector-project ready!

Name Link
Latest commit 5e066038ee1f21ea8ecb430f32fdc5dc62dc983c
Latest deploy log https://app.netlify.com/sites/vector-project/deploys/63ceb4513ba44300098dd3c4
Deploy Preview https://deploy-preview-15901--vector-project.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify[bot] avatar Jan 11 '23 13:01 netlify[bot]

Deploy Preview for vrl-playground canceled.

Name Link
Latest commit 5e066038ee1f21ea8ecb430f32fdc5dc62dc983c
Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/63ceb451b64e330008e4c357

netlify[bot] avatar Jan 11 '23 13:01 netlify[bot]

Regression Test Results

Run ID: 5f2f5e49-06aa-43d1-85fe-114d2a9a37c8
Baseline: 1727e729487c4075c29d1ca30cda5053def52085
Comparison: 801f40b6864207d7d720b2ed3b5b698ad1994574
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
syslog_loki -861.99KiB/CPU-s -9.53 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole 805.62KiB/CPU-s 2.62 100.00% 30.0MiB/CPU-s 1.66MiB/CPU-s 22.0KiB/CPU-s 0.0 0.055472 30.79MiB/CPU-s 1.14MiB/CPU-s 15.03KiB/CPU-s 0.0 0.036917 False False
splunk_hec_route_s3 45.87KiB/CPU-s 0.38 100.00% 11.74MiB/CPU-s 548.31KiB/CPU-s 7.07KiB/CPU-s 0.0 0.045595 11.79MiB/CPU-s 593.83KiB/CPU-s 7.66KiB/CPU-s 0.0 0.049193 False False
enterprise_http_to_http 10.49KiB/CPU-s 0.08 96.55% 13.61MiB/CPU-s 332.49KiB/CPU-s 4.29KiB/CPU-s 0.0 0.023852 13.62MiB/CPU-s 193.14KiB/CPU-s 2.49KiB/CPU-s 0.0 0.013845 False False
splunk_hec_to_splunk_hec_logs_noack 4.54KiB/CPU-s 0.03 75.93% 13.62MiB/CPU-s 228.72KiB/CPU-s 2.95KiB/CPU-s 0.0 0.016397 13.63MiB/CPU-s 193.32KiB/CPU-s 2.5KiB/CPU-s 0.0 0.013855 False False
http_to_http_noack 1.97KiB/CPU-s 0.01 28.44% 13.61MiB/CPU-s 307.18KiB/CPU-s 3.96KiB/CPU-s 0.0 0.022033 13.62MiB/CPU-s 286.36KiB/CPU-s 3.7KiB/CPU-s 0.0 0.020537 False False
splunk_hec_indexer_ack_blackhole 1.02KiB/CPU-s 0.01 17.95% 13.62MiB/CPU-s 251.69KiB/CPU-s 3.25KiB/CPU-s 0.0 0.018049 13.62MiB/CPU-s 243.54KiB/CPU-s 3.14KiB/CPU-s 0.0 0.017463 False False
fluent_elasticsearch -467.08B/CPU-s -0.0 59.77% 45.41MiB/CPU-s 30.22KiB/CPU-s 395.13B/CPU-s 0.0 0.00065 45.41MiB/CPU-s 30.1KiB/CPU-s 393.53B/CPU-s 0.0 0.000647 False False
http_to_http_json -138.56B/CPU-s -0.0 2.60% 13.62MiB/CPU-s 212.27KiB/CPU-s 2.74KiB/CPU-s 0.0 0.015216 13.62MiB/CPU-s 241.74KiB/CPU-s 3.12KiB/CPU-s 0.0 0.017329 False False
file_to_blackhole -3.35KiB/CPU-s -0.01 11.72% 54.5MiB/CPU-s 1.32MiB/CPU-s 17.43KiB/CPU-s 0.0 0.024214 54.5MiB/CPU-s 1.11MiB/CPU-s 14.62KiB/CPU-s 0.0 0.02033 False False
splunk_hec_to_splunk_hec_logs_acks -1.02KiB/CPU-s -0.01 12.33% 13.62MiB/CPU-s 356.31KiB/CPU-s 4.6KiB/CPU-s 0.0 0.025551 13.62MiB/CPU-s 364.46KiB/CPU-s 4.7KiB/CPU-s 0.0 0.026137 False False
http_to_http_acks -3.36KiB/CPU-s -0.06 5.23% 5.25MiB/CPU-s 2.69MiB/CPU-s 35.58KiB/CPU-s 0.0 0.512403 5.25MiB/CPU-s 2.78MiB/CPU-s 36.78KiB/CPU-s 0.0 0.530158 True False
syslog_humio_logs -47.1KiB/CPU-s -0.52 100.00% 8.87MiB/CPU-s 313.2KiB/CPU-s 4.04KiB/CPU-s 0.0 0.034471 8.83MiB/CPU-s 108.47KiB/CPU-s 1.4KiB/CPU-s 0.0 0.012 False False
datadog_agent_remap_datadog_logs_acks -218.17KiB/CPU-s -0.64 100.00% 33.54MiB/CPU-s 1.32MiB/CPU-s 17.38KiB/CPU-s 0.0 0.039218 33.33MiB/CPU-s 750.34KiB/CPU-s 9.69KiB/CPU-s 0.0 0.021985 False False
syslog_regex_logs2metric_ddmetrics -41.82KiB/CPU-s -1.11 100.00% 3.67MiB/CPU-s 385.94KiB/CPU-s 4.98KiB/CPU-s 0.0 0.102696 3.63MiB/CPU-s 399.43KiB/CPU-s 5.16KiB/CPU-s 0.0 0.107482 True False
socket_to_socket_blackhole -197.45KiB/CPU-s -1.4 100.00% 13.78MiB/CPU-s 165.37KiB/CPU-s 2.13KiB/CPU-s 0.0 0.011721 13.58MiB/CPU-s 260.87KiB/CPU-s 3.37KiB/CPU-s 0.0 0.018752 False False
otlp_http_to_blackhole -23.35KiB/CPU-s -1.48 100.00% 1.54MiB/CPU-s 114.11KiB/CPU-s 1.47KiB/CPU-s 0.0 0.072153 1.52MiB/CPU-s 126.52KiB/CPU-s 1.63KiB/CPU-s 0.0 0.081201 False False
http_text_to_http_json -487.22KiB/CPU-s -1.84 100.00% 25.86MiB/CPU-s 556.72KiB/CPU-s 7.19KiB/CPU-s 0.0 0.021022 25.38MiB/CPU-s 533.94KiB/CPU-s 6.89KiB/CPU-s 0.0 0.020539 False False
datadog_agent_remap_datadog_logs -736.74KiB/CPU-s -2.16 100.00% 33.31MiB/CPU-s 1.75MiB/CPU-s 23.1KiB/CPU-s 0.0 0.052513 32.59MiB/CPU-s 1.64MiB/CPU-s 21.69KiB/CPU-s 0.0 0.050396 False False
syslog_log2metric_humio_metrics -151.08KiB/CPU-s -2.39 100.00% 6.17MiB/CPU-s 130.06KiB/CPU-s 1.68KiB/CPU-s 0.0 0.02057 6.03MiB/CPU-s 175.39KiB/CPU-s 2.26KiB/CPU-s 0.0 0.028417 False False
datadog_agent_remap_blackhole_acks -877.21KiB/CPU-s -2.69 100.00% 31.86MiB/CPU-s 824.86KiB/CPU-s 10.65KiB/CPU-s 0.0 0.025278 31.01MiB/CPU-s 1.49MiB/CPU-s 19.73KiB/CPU-s 0.0 0.04814 False False
syslog_splunk_hec_logs -293.89KiB/CPU-s -3.16 100.00% 9.09MiB/CPU-s 75.36KiB/CPU-s 996.19B/CPU-s 0.0 0.008099 8.8MiB/CPU-s 176.93KiB/CPU-s 2.28KiB/CPU-s 0.0 0.019635 False False
otlp_grpc_to_blackhole -37.04KiB/CPU-s -3.45 100.00% 1.05MiB/CPU-s 40.4KiB/CPU-s 533.98B/CPU-s 0.0 0.037654 1.01MiB/CPU-s 51.18KiB/CPU-s 676.3B/CPU-s 0.0 0.049403 False False
syslog_log2metric_splunk_hec_metrics -437.83KiB/CPU-s -4.61 100.00% 9.28MiB/CPU-s 342.11KiB/CPU-s 4.42KiB/CPU-s 0.0 0.036008 8.85MiB/CPU-s 433.58KiB/CPU-s 5.6KiB/CPU-s 0.0 0.047841 False False
syslog_loki -861.99KiB/CPU-s -9.53 100.00% 8.83MiB/CPU-s 218.94KiB/CPU-s 2.83KiB/CPU-s 0.0 0.024211 7.99MiB/CPU-s 360.04KiB/CPU-s 4.65KiB/CPU-s 0.0 0.04401 False False

github-actions[bot] avatar Jan 11 '23 14:01 github-actions[bot]

Regression Test Results

Run ID: db94b8fb-5c98-4f7d-b4c7-7fb13c8adaa2
Baseline: c66dd2a5f6700de1ba4e9ca84fed2f330f0f018a
Comparison: 896ca273100b96d2106b733a0bf0a72aff372323
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole_acks 1.03MiB/CPU-s 3.38 100.00% 30.64MiB/CPU-s 1.93MiB/CPU-s 25.46KiB/CPU-s 0.0 0.062825 31.68MiB/CPU-s 863.53KiB/CPU-s 11.15KiB/CPU-s 0.0 0.026619 False False
http_text_to_http_json 814.49KiB/CPU-s 3.31 100.00% 24.04MiB/CPU-s 814.25KiB/CPU-s 10.56KiB/CPU-s 0.0 0.033079 24.83MiB/CPU-s 614.38KiB/CPU-s 7.93KiB/CPU-s 0.0 0.02416 False False
datadog_agent_remap_datadog_logs 1.1MiB/CPU-s 3.29 100.00% 33.36MiB/CPU-s 1.78MiB/CPU-s 23.54KiB/CPU-s 0.0 0.053415 34.46MiB/CPU-s 1.14MiB/CPU-s 15.02KiB/CPU-s 0.0 0.033005 False False
syslog_regex_logs2metric_ddmetrics 85.86KiB/CPU-s 2.26 100.00% 3.71MiB/CPU-s 427.24KiB/CPU-s 5.52KiB/CPU-s 0.0 0.112467 3.79MiB/CPU-s 460.79KiB/CPU-s 5.95KiB/CPU-s 0.0 0.118616 True False
syslog_splunk_hec_logs 158.58KiB/CPU-s 1.73 100.00% 8.93MiB/CPU-s 254.07KiB/CPU-s 3.28KiB/CPU-s 0.0 0.02778 9.09MiB/CPU-s 193.25KiB/CPU-s 2.49KiB/CPU-s 0.0 0.02077 False False
syslog_humio_logs 71.36KiB/CPU-s 0.77 100.00% 9.05MiB/CPU-s 308.51KiB/CPU-s 3.98KiB/CPU-s 0.0 0.033296 9.12MiB/CPU-s 129.59KiB/CPU-s 1.67KiB/CPU-s 0.0 0.013879 False False
syslog_log2metric_humio_metrics 43.13KiB/CPU-s 0.68 100.00% 6.22MiB/CPU-s 75.86KiB/CPU-s 1002.87B/CPU-s 0.0 0.011912 6.26MiB/CPU-s 127.0KiB/CPU-s 1.64KiB/CPU-s 0.0 0.019808 False False
datadog_agent_remap_datadog_logs_acks 168.04KiB/CPU-s 0.49 100.00% 33.68MiB/CPU-s 1.14MiB/CPU-s 15.09KiB/CPU-s 0.0 0.033911 33.85MiB/CPU-s 1.0MiB/CPU-s 13.27KiB/CPU-s 0.0 0.029663 False False
socket_to_socket_blackhole 66.78KiB/CPU-s 0.49 100.00% 13.38MiB/CPU-s 168.79KiB/CPU-s 2.18KiB/CPU-s 0.0 0.012316 13.45MiB/CPU-s 227.66KiB/CPU-s 2.94KiB/CPU-s 0.0 0.016531 False False
http_to_http_acks 6.06KiB/CPU-s 0.11 9.32% 5.26MiB/CPU-s 2.78MiB/CPU-s 36.77KiB/CPU-s 0.0 0.528655 5.27MiB/CPU-s 2.76MiB/CPU-s 36.41KiB/CPU-s 0.0 0.522998 True False
enterprise_http_to_http 14.58KiB/CPU-s 0.1 99.69% 13.61MiB/CPU-s 351.57KiB/CPU-s 4.54KiB/CPU-s 0.0 0.025225 13.62MiB/CPU-s 149.69KiB/CPU-s 1.93KiB/CPU-s 0.0 0.010729 False False
syslog_loki 3.52KiB/CPU-s 0.04 58.44% 8.85MiB/CPU-s 220.67KiB/CPU-s 2.85KiB/CPU-s 0.0 0.024356 8.85MiB/CPU-s 251.62KiB/CPU-s 3.25KiB/CPU-s 0.0 0.02776 False False
otlp_grpc_to_blackhole 485.97B/CPU-s 0.04 42.82% 1.04MiB/CPU-s 37.65KiB/CPU-s 497.72B/CPU-s 0.0 0.035188 1.05MiB/CPU-s 53.05KiB/CPU-s 700.74B/CPU-s 0.0 0.049552 False False
splunk_hec_route_s3 3.82KiB/CPU-s 0.03 30.72% 11.76MiB/CPU-s 540.5KiB/CPU-s 6.97KiB/CPU-s 0.0 0.044885 11.76MiB/CPU-s 518.0KiB/CPU-s 6.69KiB/CPU-s 0.0 0.043004 False False
splunk_hec_to_splunk_hec_logs_noack 2.39KiB/CPU-s 0.02 48.25% 13.62MiB/CPU-s 212.02KiB/CPU-s 2.74KiB/CPU-s 0.0 0.015197 13.63MiB/CPU-s 191.86KiB/CPU-s 2.48KiB/CPU-s 0.0 0.01375 False False
http_to_http_noack 2.12KiB/CPU-s 0.02 30.34% 13.61MiB/CPU-s 310.14KiB/CPU-s 4.0KiB/CPU-s 0.0 0.022246 13.62MiB/CPU-s 285.26KiB/CPU-s 3.68KiB/CPU-s 0.0 0.020458 False False
fluent_elasticsearch -380.39B/CPU-s -0.0 51.07% 45.41MiB/CPU-s 29.38KiB/CPU-s 384.03B/CPU-s 0.0 0.000632 45.41MiB/CPU-s 30.13KiB/CPU-s 394.0B/CPU-s 0.0 0.000648 False False
http_to_http_json 69.69B/CPU-s 0.0 1.41% 13.62MiB/CPU-s 211.05KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015129 13.62MiB/CPU-s 210.76KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015108 False False
file_to_blackhole -7.05KiB/CPU-s -0.01 27.12% 54.5MiB/CPU-s 1.04MiB/CPU-s 13.72KiB/CPU-s 0.0 0.019059 54.49MiB/CPU-s 1.14MiB/CPU-s 15.01KiB/CPU-s 0.0 0.020871 False False
splunk_hec_indexer_ack_blackhole -744.46B/CPU-s -0.01 12.73% 13.62MiB/CPU-s 246.5KiB/CPU-s 3.18KiB/CPU-s 0.0 0.017675 13.62MiB/CPU-s 250.83KiB/CPU-s 3.24KiB/CPU-s 0.0 0.017987 False False
splunk_hec_to_splunk_hec_logs_acks -849.95B/CPU-s -0.01 10.69% 13.62MiB/CPU-s 334.29KiB/CPU-s 4.31KiB/CPU-s 0.0 0.023972 13.62MiB/CPU-s 342.94KiB/CPU-s 4.42KiB/CPU-s 0.0 0.024594 False False
otlp_http_to_blackhole -2.0KiB/CPU-s -0.13 65.35% 1.54MiB/CPU-s 113.51KiB/CPU-s 1.46KiB/CPU-s 0.0 0.072201 1.53MiB/CPU-s 119.28KiB/CPU-s 1.54KiB/CPU-s 0.0 0.075965 False False
syslog_log2metric_splunk_hec_metrics -12.84KiB/CPU-s -0.13 99.97% 9.32MiB/CPU-s 200.94KiB/CPU-s 2.59KiB/CPU-s 0.0 0.021044 9.31MiB/CPU-s 184.78KiB/CPU-s 2.39KiB/CPU-s 0.0 0.019379 False False
datadog_agent_remap_blackhole -284.4KiB/CPU-s -0.89 100.00% 31.12MiB/CPU-s 896.92KiB/CPU-s 11.58KiB/CPU-s 0.0 0.028143 30.84MiB/CPU-s 989.65KiB/CPU-s 12.77KiB/CPU-s 0.0 0.031333 False False

github-actions[bot] avatar Jan 16 '23 19:01 github-actions[bot]

Regression Test Results

Run ID: 41af5065-a48f-4716-80c8-78a9e80a2e06
Baseline: 889c167b78c93c2de8b02576f65608e128071110
Comparison: b3e55c1a07a79458e79dab4cca19145dffececb5
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_humio_metrics 271.96KiB/CPU-s 4.38 100.00% 6.06MiB/CPU-s 239.44KiB/CPU-s 3.09KiB/CPU-s 0.0 0.038599 6.32MiB/CPU-s 178.64KiB/CPU-s 2.31KiB/CPU-s 0.0 0.027588 False False
http_text_to_http_json 834.53KiB/CPU-s 3.45 100.00% 23.63MiB/CPU-s 771.61KiB/CPU-s 9.96KiB/CPU-s 0.0 0.031882 24.45MiB/CPU-s 544.4KiB/CPU-s 7.02KiB/CPU-s 0.0 0.021744 False False
syslog_splunk_hec_logs 206.76KiB/CPU-s 2.3 100.00% 8.78MiB/CPU-s 270.34KiB/CPU-s 3.49KiB/CPU-s 0.0 0.030056 8.98MiB/CPU-s 242.65KiB/CPU-s 3.13KiB/CPU-s 0.0 0.026371 False False
syslog_log2metric_splunk_hec_metrics 191.03KiB/CPU-s 2.03 100.00% 9.2MiB/CPU-s 337.5KiB/CPU-s 4.36KiB/CPU-s 0.0 0.035807 9.39MiB/CPU-s 216.82KiB/CPU-s 2.8KiB/CPU-s 0.0 0.022547 False False
otlp_http_to_blackhole 29.33KiB/CPU-s 1.89 100.00% 1.51MiB/CPU-s 131.66KiB/CPU-s 1.7KiB/CPU-s 0.0 0.084911 1.54MiB/CPU-s 117.95KiB/CPU-s 1.52KiB/CPU-s 0.0 0.074656 False False
datadog_agent_remap_blackhole_acks 474.81KiB/CPU-s 1.5 100.00% 30.99MiB/CPU-s 1.49MiB/CPU-s 19.74KiB/CPU-s 0.0 0.048178 31.45MiB/CPU-s 1.01MiB/CPU-s 13.35KiB/CPU-s 0.0 0.032118 False False
socket_to_socket_blackhole 104.5KiB/CPU-s 0.76 100.00% 13.39MiB/CPU-s 199.8KiB/CPU-s 2.58KiB/CPU-s 0.0 0.014572 13.49MiB/CPU-s 149.0KiB/CPU-s 1.92KiB/CPU-s 0.0 0.010785 False False
syslog_humio_logs 45.67KiB/CPU-s 0.51 100.00% 8.79MiB/CPU-s 496.78KiB/CPU-s 6.41KiB/CPU-s 0.0 0.055203 8.83MiB/CPU-s 499.79KiB/CPU-s 6.45KiB/CPU-s 0.0 0.055257 False False
datadog_agent_remap_datadog_logs_acks 119.55KiB/CPU-s 0.35 100.00% 33.21MiB/CPU-s 1.22MiB/CPU-s 16.09KiB/CPU-s 0.0 0.036661 33.33MiB/CPU-s 1.15MiB/CPU-s 15.25KiB/CPU-s 0.0 0.03462 False False
splunk_hec_to_splunk_hec_logs_noack 6.51KiB/CPU-s 0.05 89.80% 13.62MiB/CPU-s 241.07KiB/CPU-s 3.11KiB/CPU-s 0.0 0.017285 13.63MiB/CPU-s 192.3KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013782 False False
splunk_hec_route_s3 4.17KiB/CPU-s 0.04 29.92% 11.48MiB/CPU-s 624.35KiB/CPU-s 8.05KiB/CPU-s 0.0 0.053092 11.49MiB/CPU-s 563.74KiB/CPU-s 7.28KiB/CPU-s 0.0 0.04792 False False
enterprise_http_to_http 4.06KiB/CPU-s 0.03 67.19% 13.62MiB/CPU-s 256.49KiB/CPU-s 3.31KiB/CPU-s 0.0 0.018392 13.62MiB/CPU-s 194.23KiB/CPU-s 2.51KiB/CPU-s 0.0 0.013923 False False
splunk_hec_to_splunk_hec_logs_acks 2.27KiB/CPU-s 0.02 27.38% 13.61MiB/CPU-s 355.58KiB/CPU-s 4.59KiB/CPU-s 0.0 0.025505 13.62MiB/CPU-s 354.8KiB/CPU-s 4.58KiB/CPU-s 0.0 0.025445 False False
http_to_http_json -279.4B/CPU-s -0.0 5.67% 13.62MiB/CPU-s 209.72KiB/CPU-s 2.71KiB/CPU-s 0.0 0.015033 13.62MiB/CPU-s 211.05KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015129 False False
fluent_elasticsearch -68.97B/CPU-s -0.0 9.87% 45.41MiB/CPU-s 30.02KiB/CPU-s 392.55B/CPU-s 0.0 0.000645 45.41MiB/CPU-s 30.12KiB/CPU-s 393.77B/CPU-s 0.0 0.000648 False False
file_to_blackhole -5.32KiB/CPU-s -0.01 17.12% 54.48MiB/CPU-s 1.37MiB/CPU-s 18.06KiB/CPU-s 0.0 0.025101 54.48MiB/CPU-s 1.27MiB/CPU-s 16.73KiB/CPU-s 0.0 0.023262 False False
splunk_hec_indexer_ack_blackhole -1.11KiB/CPU-s -0.01 18.82% 13.62MiB/CPU-s 251.85KiB/CPU-s 3.25KiB/CPU-s 0.0 0.018061 13.62MiB/CPU-s 258.04KiB/CPU-s 3.33KiB/CPU-s 0.0 0.018506 False False
http_to_http_noack -6.08KiB/CPU-s -0.04 74.17% 13.62MiB/CPU-s 258.4KiB/CPU-s 3.34KiB/CPU-s 0.0 0.018529 13.61MiB/CPU-s 326.86KiB/CPU-s 4.22KiB/CPU-s 0.0 0.023448 False False
datadog_agent_remap_blackhole -13.49KiB/CPU-s -0.04 42.35% 31.34MiB/CPU-s 1.28MiB/CPU-s 16.94KiB/CPU-s 0.0 0.040875 31.33MiB/CPU-s 1.3MiB/CPU-s 17.21KiB/CPU-s 0.0 0.041576 False False
http_to_http_acks -3.52KiB/CPU-s -0.07 5.53% 5.19MiB/CPU-s 2.71MiB/CPU-s 35.79KiB/CPU-s 0.0 0.521332 5.19MiB/CPU-s 2.72MiB/CPU-s 35.94KiB/CPU-s 0.0 0.523866 True False
otlp_grpc_to_blackhole -1.36KiB/CPU-s -0.13 88.78% 1.05MiB/CPU-s 44.36KiB/CPU-s 586.26B/CPU-s 0.0 0.041395 1.05MiB/CPU-s 49.54KiB/CPU-s 654.53B/CPU-s 0.0 0.046282 False False
datadog_agent_remap_datadog_logs -68.92KiB/CPU-s -0.2 99.84% 34.3MiB/CPU-s 1.11MiB/CPU-s 14.69KiB/CPU-s 0.0 0.032432 34.23MiB/CPU-s 1.23MiB/CPU-s 16.18KiB/CPU-s 0.0 0.035791 False False
syslog_loki -75.19KiB/CPU-s -0.82 100.00% 8.95MiB/CPU-s 167.77KiB/CPU-s 2.17KiB/CPU-s 0.0 0.018301 8.88MiB/CPU-s 141.63KiB/CPU-s 1.83KiB/CPU-s 0.0 0.015577 False False
syslog_regex_logs2metric_ddmetrics -64.47KiB/CPU-s -1.68 100.00% 3.74MiB/CPU-s 452.89KiB/CPU-s 5.84KiB/CPU-s 0.0 0.118275 3.68MiB/CPU-s 485.41KiB/CPU-s 6.27KiB/CPU-s 0.0 0.128939 True False

github-actions[bot] avatar Jan 16 '23 19:01 github-actions[bot]

Regression Test Results

Run ID: 422fe2ea-8ce7-4638-8d78-fb4b3b59c37b
Baseline: e12a08ca7bd2d6f1e0cadecad7616cd4adf587ee
Comparison: adb5b0b2896aaaf0f7c1745902eeed61afee0ac4
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json 796.16KiB/CPU-s 3.23 100.00% 24.08MiB/CPU-s 498.26KiB/CPU-s 6.43KiB/CPU-s 0.0 0.020207 24.86MiB/CPU-s 548.26KiB/CPU-s 7.07KiB/CPU-s 0.0 0.021539 False False
datadog_agent_remap_datadog_logs 836.6KiB/CPU-s 2.5 100.00% 32.68MiB/CPU-s 1.47MiB/CPU-s 19.42KiB/CPU-s 0.0 0.044976 33.5MiB/CPU-s 1.66MiB/CPU-s 21.91KiB/CPU-s 0.0 0.049516 False False
datadog_agent_remap_datadog_logs_acks 778.73KiB/CPU-s 2.28 100.00% 33.3MiB/CPU-s 1.48MiB/CPU-s 19.57KiB/CPU-s 0.0 0.044469 34.06MiB/CPU-s 1.13MiB/CPU-s 14.95KiB/CPU-s 0.0 0.033209 False False
otlp_http_to_blackhole 32.47KiB/CPU-s 2.08 100.00% 1.52MiB/CPU-s 118.13KiB/CPU-s 1.52KiB/CPU-s 0.0 0.075722 1.56MiB/CPU-s 109.38KiB/CPU-s 1.41KiB/CPU-s 0.0 0.068685 False False
syslog_regex_logs2metric_ddmetrics 70.03KiB/CPU-s 1.85 100.00% 3.7MiB/CPU-s 430.73KiB/CPU-s 5.56KiB/CPU-s 0.0 0.113773 3.77MiB/CPU-s 461.9KiB/CPU-s 5.96KiB/CPU-s 0.0 0.11979 True False
otlp_grpc_to_blackhole 14.46KiB/CPU-s 1.36 100.00% 1.04MiB/CPU-s 40.39KiB/CPU-s 533.9B/CPU-s 0.0 0.038044 1.05MiB/CPU-s 49.51KiB/CPU-s 654.2B/CPU-s 0.0 0.046009 False False
socket_to_socket_blackhole 174.0KiB/CPU-s 1.29 100.00% 13.21MiB/CPU-s 323.15KiB/CPU-s 4.17KiB/CPU-s 0.0 0.023888 13.38MiB/CPU-s 373.37KiB/CPU-s 4.82KiB/CPU-s 0.0 0.02725 False False
syslog_log2metric_humio_metrics 50.95KiB/CPU-s 0.8 100.00% 6.18MiB/CPU-s 170.66KiB/CPU-s 2.2KiB/CPU-s 0.0 0.026951 6.23MiB/CPU-s 280.71KiB/CPU-s 3.62KiB/CPU-s 0.0 0.043975 False False
enterprise_http_to_http 18.51KiB/CPU-s 0.13 99.94% 13.61MiB/CPU-s 390.15KiB/CPU-s 5.03KiB/CPU-s 0.0 0.028001 13.62MiB/CPU-s 148.62KiB/CPU-s 1.92KiB/CPU-s 0.0 0.010652 False False
splunk_hec_to_splunk_hec_logs_acks 4.22KiB/CPU-s 0.03 50.72% 13.61MiB/CPU-s 343.81KiB/CPU-s 4.43KiB/CPU-s 0.0 0.024662 13.62MiB/CPU-s 331.08KiB/CPU-s 4.27KiB/CPU-s 0.0 0.023742 False False
splunk_hec_to_splunk_hec_logs_noack 3.33KiB/CPU-s 0.02 62.11% 13.62MiB/CPU-s 221.83KiB/CPU-s 2.86KiB/CPU-s 0.0 0.015902 13.63MiB/CPU-s 192.37KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013786 False False
http_to_http_noack 1.72KiB/CPU-s 0.01 25.14% 13.61MiB/CPU-s 302.65KiB/CPU-s 3.91KiB/CPU-s 0.0 0.021708 13.62MiB/CPU-s 286.56KiB/CPU-s 3.7KiB/CPU-s 0.0 0.020551 False False
fluent_elasticsearch 140.74B/CPU-s 0.0 20.15% 45.41MiB/CPU-s 30.0KiB/CPU-s 392.37B/CPU-s 0.0 0.000645 45.41MiB/CPU-s 29.62KiB/CPU-s 387.34B/CPU-s 0.0 0.000637 False False
file_to_blackhole 239.61B/CPU-s 0.0 0.88% 54.5MiB/CPU-s 1.1MiB/CPU-s 14.56KiB/CPU-s 0.0 0.02023 54.5MiB/CPU-s 1.17MiB/CPU-s 15.49KiB/CPU-s 0.0 0.021527 False False
splunk_hec_indexer_ack_blackhole -1.76KiB/CPU-s -0.01 29.97% 13.62MiB/CPU-s 247.46KiB/CPU-s 3.19KiB/CPU-s 0.0 0.017744 13.62MiB/CPU-s 254.01KiB/CPU-s 3.28KiB/CPU-s 0.0 0.018216 False False
http_to_http_json -6.66KiB/CPU-s -0.05 88.04% 13.62MiB/CPU-s 228.62KiB/CPU-s 2.95KiB/CPU-s 0.0 0.016397 13.61MiB/CPU-s 240.14KiB/CPU-s 3.1KiB/CPU-s 0.0 0.017231 False False
http_to_http_acks -15.28KiB/CPU-s -0.28 22.92% 5.27MiB/CPU-s 2.75MiB/CPU-s 36.3KiB/CPU-s 0.0 0.521025 5.26MiB/CPU-s 2.86MiB/CPU-s 37.83KiB/CPU-s 0.0 0.544432 True False
syslog_splunk_hec_logs -39.36KiB/CPU-s -0.43 100.00% 8.85MiB/CPU-s 260.02KiB/CPU-s 3.36KiB/CPU-s 0.0 0.028693 8.81MiB/CPU-s 328.85KiB/CPU-s 4.24KiB/CPU-s 0.0 0.036446 False False
syslog_loki -42.35KiB/CPU-s -0.46 100.00% 8.95MiB/CPU-s 160.66KiB/CPU-s 2.07KiB/CPU-s 0.0 0.017525 8.91MiB/CPU-s 152.31KiB/CPU-s 1.97KiB/CPU-s 0.0 0.016691 False False
datadog_agent_remap_blackhole -308.38KiB/CPU-s -0.98 100.00% 30.74MiB/CPU-s 1.29MiB/CPU-s 17.09KiB/CPU-s 0.0 0.042064 30.44MiB/CPU-s 1.49MiB/CPU-s 19.68KiB/CPU-s 0.0 0.048919 False False
syslog_log2metric_splunk_hec_metrics -104.71KiB/CPU-s -1.11 100.00% 9.19MiB/CPU-s 374.92KiB/CPU-s 4.84KiB/CPU-s 0.0 0.039848 9.09MiB/CPU-s 486.58KiB/CPU-s 6.28KiB/CPU-s 0.0 0.052297 False False
splunk_hec_route_s3 -141.29KiB/CPU-s -1.17 100.00% 11.84MiB/CPU-s 521.81KiB/CPU-s 6.73KiB/CPU-s 0.0 0.043033 11.7MiB/CPU-s 593.22KiB/CPU-s 7.66KiB/CPU-s 0.0 0.049499 False False
syslog_humio_logs -149.27KiB/CPU-s -1.59 100.00% 9.18MiB/CPU-s 262.07KiB/CPU-s 3.38KiB/CPU-s 0.0 0.027885 9.03MiB/CPU-s 385.98KiB/CPU-s 4.98KiB/CPU-s 0.0 0.041731 False False
datadog_agent_remap_blackhole_acks -1.26MiB/CPU-s -4.11 100.00% 30.74MiB/CPU-s 1.58MiB/CPU-s 20.9KiB/CPU-s 0.0 0.051432 29.48MiB/CPU-s 2.3MiB/CPU-s 30.42KiB/CPU-s 0.0 0.078077 False False

github-actions[bot] avatar Jan 17 '23 17:01 github-actions[bot]

null! image

spencergilbert avatar Jan 17 '23 18:01 spencergilbert

Regression Test Results

Run ID: abee4c17-1d5b-49ef-b8f2-d1a6b7b6273f
Baseline: c6d81006ec03aa0db709559d5e08846f7a3b6caa
Comparison: 56c1b3d2afd155a5fb8e5ff1ff33b0b489dbb1fc
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
fluent_elasticsearch -3.83MiB/CPU-s -8.44 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_splunk_hec_metrics 416.06KiB/CPU-s 4.52 100.00% 8.99MiB/CPU-s 1.17MiB/CPU-s 15.51KiB/CPU-s 0.0 0.130421 9.4MiB/CPU-s 208.17KiB/CPU-s 2.69KiB/CPU-s 0.0 0.021623 True False
datadog_agent_remap_datadog_logs_acks 382.64KiB/CPU-s 1.11 100.00% 33.69MiB/CPU-s 1.19MiB/CPU-s 15.71KiB/CPU-s 0.0 0.035271 34.07MiB/CPU-s 915.82KiB/CPU-s 11.82KiB/CPU-s 0.0 0.026251 False False
datadog_agent_remap_blackhole 268.3KiB/CPU-s 0.85 100.00% 30.92MiB/CPU-s 1.17MiB/CPU-s 15.48KiB/CPU-s 0.0 0.03789 31.19MiB/CPU-s 941.29KiB/CPU-s 12.15KiB/CPU-s 0.0 0.029474 False False
otlp_grpc_to_blackhole 7.83KiB/CPU-s 0.73 100.00% 1.04MiB/CPU-s 44.87KiB/CPU-s 592.99B/CPU-s 0.0 0.041969 1.05MiB/CPU-s 48.41KiB/CPU-s 639.63B/CPU-s 0.0 0.044952 False False
syslog_splunk_hec_logs 64.76KiB/CPU-s 0.71 100.00% 8.97MiB/CPU-s 97.23KiB/CPU-s 1.26KiB/CPU-s 0.0 0.010584 9.03MiB/CPU-s 189.93KiB/CPU-s 2.45KiB/CPU-s 0.0 0.020532 False False
syslog_loki 34.86KiB/CPU-s 0.39 100.00% 8.77MiB/CPU-s 225.86KiB/CPU-s 2.91KiB/CPU-s 0.0 0.025144 8.81MiB/CPU-s 258.75KiB/CPU-s 3.34KiB/CPU-s 0.0 0.028693 False False
syslog_regex_logs2metric_ddmetrics 12.17KiB/CPU-s 0.33 92.00% 3.58MiB/CPU-s 342.59KiB/CPU-s 4.42KiB/CPU-s 0.0 0.093552 3.59MiB/CPU-s 415.41KiB/CPU-s 5.36KiB/CPU-s 0.0 0.113061 True False
datadog_agent_remap_datadog_logs 86.05KiB/CPU-s 0.25 100.00% 34.27MiB/CPU-s 1.16MiB/CPU-s 15.32KiB/CPU-s 0.0 0.033842 34.36MiB/CPU-s 1.09MiB/CPU-s 14.46KiB/CPU-s 0.0 0.031848 False False
enterprise_http_to_http 5.95KiB/CPU-s 0.04 82.15% 13.62MiB/CPU-s 284.57KiB/CPU-s 3.67KiB/CPU-s 0.0 0.020408 13.62MiB/CPU-s 190.42KiB/CPU-s 2.46KiB/CPU-s 0.0 0.01365 False False
splunk_hec_to_splunk_hec_logs_noack 5.3KiB/CPU-s 0.04 83.08% 13.62MiB/CPU-s 229.53KiB/CPU-s 2.96KiB/CPU-s 0.0 0.016456 13.63MiB/CPU-s 191.05KiB/CPU-s 2.47KiB/CPU-s 0.0 0.013692 False False
splunk_hec_indexer_ack_blackhole 1.17KiB/CPU-s 0.01 20.66% 13.62MiB/CPU-s 246.93KiB/CPU-s 3.19KiB/CPU-s 0.0 0.017706 13.62MiB/CPU-s 242.16KiB/CPU-s 3.12KiB/CPU-s 0.0 0.017363 False False
splunk_hec_to_splunk_hec_logs_acks 1.21KiB/CPU-s 0.01 15.20% 13.62MiB/CPU-s 348.2KiB/CPU-s 4.49KiB/CPU-s 0.0 0.024972 13.62MiB/CPU-s 342.34KiB/CPU-s 4.42KiB/CPU-s 0.0 0.02455 False False
http_to_http_json 53.52B/CPU-s 0.0 1.08% 13.62MiB/CPU-s 210.62KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015098 13.62MiB/CPU-s 211.39KiB/CPU-s 2.73KiB/CPU-s 0.0 0.015153 False False
http_to_http_noack 17.81B/CPU-s 0.0 0.23% 13.61MiB/CPU-s 332.7KiB/CPU-s 4.29KiB/CPU-s 0.0 0.023867 13.61MiB/CPU-s 331.3KiB/CPU-s 4.27KiB/CPU-s 0.0 0.023767 False False
file_to_blackhole -16.33KiB/CPU-s -0.03 53.71% 54.49MiB/CPU-s 1.12MiB/CPU-s 14.73KiB/CPU-s 0.0 0.020466 54.48MiB/CPU-s 1.26MiB/CPU-s 16.67KiB/CPU-s 0.0 0.02319 False False
splunk_hec_route_s3 -11.95KiB/CPU-s -0.1 77.17% 11.81MiB/CPU-s 560.88KiB/CPU-s 7.23KiB/CPU-s 0.0 0.046393 11.79MiB/CPU-s 526.2KiB/CPU-s 6.79KiB/CPU-s 0.0 0.043568 False False
datadog_agent_remap_blackhole_acks -57.42KiB/CPU-s -0.18 98.98% 31.33MiB/CPU-s 1.28MiB/CPU-s 16.95KiB/CPU-s 0.0 0.040894 31.28MiB/CPU-s 1.1MiB/CPU-s 14.55KiB/CPU-s 0.0 0.035202 False False
syslog_humio_logs -33.52KiB/CPU-s -0.36 100.00% 9.18MiB/CPU-s 92.38KiB/CPU-s 1.19KiB/CPU-s 0.0 0.009826 9.15MiB/CPU-s 108.53KiB/CPU-s 1.4KiB/CPU-s 0.0 0.011586 False False
otlp_http_to_blackhole -7.44KiB/CPU-s -0.47 99.98% 1.56MiB/CPU-s 102.29KiB/CPU-s 1.32KiB/CPU-s 0.0 0.064013 1.55MiB/CPU-s 113.69KiB/CPU-s 1.47KiB/CPU-s 0.0 0.071476 False False
syslog_log2metric_humio_metrics -29.87KiB/CPU-s -0.48 100.00% 6.14MiB/CPU-s 142.07KiB/CPU-s 1.83KiB/CPU-s 0.0 0.022609 6.11MiB/CPU-s 255.1KiB/CPU-s 3.29KiB/CPU-s 0.0 0.04079 False False
http_to_http_acks -59.39KiB/CPU-s -1.1 74.89% 5.27MiB/CPU-s 2.76MiB/CPU-s 36.5KiB/CPU-s 0.0 0.523899 5.21MiB/CPU-s 2.77MiB/CPU-s 36.67KiB/CPU-s 0.0 0.532243 True False
http_text_to_http_json -296.59KiB/CPU-s -1.16 100.00% 24.94MiB/CPU-s 582.02KiB/CPU-s 7.51KiB/CPU-s 0.0 0.022788 24.65MiB/CPU-s 585.89KiB/CPU-s 7.56KiB/CPU-s 0.0 0.023209 False False
socket_to_socket_blackhole -172.67KiB/CPU-s -1.24 100.00% 13.63MiB/CPU-s 328.15KiB/CPU-s 4.24KiB/CPU-s 0.0 0.023516 13.46MiB/CPU-s 165.17KiB/CPU-s 2.13KiB/CPU-s 0.0 0.011985 False False
fluent_elasticsearch -3.83MiB/CPU-s -8.44 100.00% 45.41MiB/CPU-s 29.98KiB/CPU-s 391.97B/CPU-s 0.0 0.000645 41.58MiB/CPU-s 11.46MiB/CPU-s 149.7KiB/CPU-s 0.0 0.27548 True False

github-actions[bot] avatar Jan 20 '23 11:01 github-actions[bot]

Regression Test Results

Run ID: 7143bc21-5ae4-4204-86f4-e6b5674f5761
Baseline: eeb2bb5f846df4c22fc2a0180d387538249bacc2
Comparison: 900f277206b4a389808f0fd14f46189e0ea0e478
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
otlp_http_to_blackhole 20.53KiB/CPU-s 1.32 100.00% 1.52MiB/CPU-s 118.73KiB/CPU-s 1.53KiB/CPU-s 0.0 0.076526 1.54MiB/CPU-s 120.28KiB/CPU-s 1.55KiB/CPU-s 0.0 0.076513 False False
http_text_to_http_json 265.21KiB/CPU-s 1.07 100.00% 24.17MiB/CPU-s 898.15KiB/CPU-s 11.59KiB/CPU-s 0.0 0.036282 24.43MiB/CPU-s 641.54KiB/CPU-s 8.28KiB/CPU-s 0.0 0.025641 False False
syslog_loki 68.16KiB/CPU-s 0.77 100.00% 8.66MiB/CPU-s 374.25KiB/CPU-s 4.83KiB/CPU-s 0.0 0.042188 8.73MiB/CPU-s 240.94KiB/CPU-s 3.11KiB/CPU-s 0.0 0.026953 False False
syslog_humio_logs 65.77KiB/CPU-s 0.71 100.00% 9.06MiB/CPU-s 320.75KiB/CPU-s 4.14KiB/CPU-s 0.0 0.034583 9.12MiB/CPU-s 152.69KiB/CPU-s 1.97KiB/CPU-s 0.0 0.016347 False False
syslog_log2metric_splunk_hec_metrics 58.92KiB/CPU-s 0.61 100.00% 9.36MiB/CPU-s 273.99KiB/CPU-s 3.54KiB/CPU-s 0.0 0.028599 9.41MiB/CPU-s 232.8KiB/CPU-s 3.01KiB/CPU-s 0.0 0.02415 False False
socket_to_socket_blackhole 79.87KiB/CPU-s 0.58 100.00% 13.4MiB/CPU-s 198.53KiB/CPU-s 2.56KiB/CPU-s 0.0 0.014472 13.47MiB/CPU-s 122.03KiB/CPU-s 1.58KiB/CPU-s 0.0 0.008844 False False
http_to_http_acks 28.25KiB/CPU-s 0.53 41.74% 5.25MiB/CPU-s 2.72MiB/CPU-s 36.0KiB/CPU-s 0.0 0.519196 5.27MiB/CPU-s 2.78MiB/CPU-s 36.69KiB/CPU-s 0.0 0.526528 True False
otlp_grpc_to_blackhole 3.56KiB/CPU-s 0.34 99.99% 1.04MiB/CPU-s 43.19KiB/CPU-s 570.88B/CPU-s 0.0 0.040726 1.04MiB/CPU-s 56.71KiB/CPU-s 749.22B/CPU-s 0.0 0.053296 False False
enterprise_http_to_http 10.26KiB/CPU-s 0.07 98.05% 13.61MiB/CPU-s 305.99KiB/CPU-s 3.95KiB/CPU-s 0.0 0.021948 13.62MiB/CPU-s 148.84KiB/CPU-s 1.92KiB/CPU-s 0.0 0.010668 False False
splunk_hec_to_splunk_hec_logs_noack 2.71KiB/CPU-s 0.02 51.72% 13.62MiB/CPU-s 228.87KiB/CPU-s 2.95KiB/CPU-s 0.0 0.016408 13.62MiB/CPU-s 193.28KiB/CPU-s 2.5KiB/CPU-s 0.0 0.013854 False False
http_to_http_noack -4.34B/CPU-s -0.0 0.07% 13.62MiB/CPU-s 285.22KiB/CPU-s 3.68KiB/CPU-s 0.0 0.020455 13.62MiB/CPU-s 284.11KiB/CPU-s 3.67KiB/CPU-s 0.0 0.020376 False False
fluent_elasticsearch -22.62B/CPU-s -0.0 3.20% 45.41MiB/CPU-s 30.43KiB/CPU-s 397.64B/CPU-s 0.0 0.000654 45.41MiB/CPU-s 30.51KiB/CPU-s 398.75B/CPU-s 0.0 0.000656 False False
file_to_blackhole -1.6KiB/CPU-s -0.0 6.22% 54.5MiB/CPU-s 1.08MiB/CPU-s 14.29KiB/CPU-s 0.0 0.01985 54.49MiB/CPU-s 1.12MiB/CPU-s 14.8KiB/CPU-s 0.0 0.02057 False False
splunk_hec_indexer_ack_blackhole -1.49KiB/CPU-s -0.01 25.53% 13.62MiB/CPU-s 248.24KiB/CPU-s 3.2KiB/CPU-s 0.0 0.017799 13.62MiB/CPU-s 253.81KiB/CPU-s 3.27KiB/CPU-s 0.0 0.018201 False False
splunk_hec_to_splunk_hec_logs_acks -1.89KiB/CPU-s -0.01 23.76% 13.62MiB/CPU-s 331.79KiB/CPU-s 4.28KiB/CPU-s 0.0 0.023791 13.62MiB/CPU-s 354.11KiB/CPU-s 4.57KiB/CPU-s 0.0 0.025395 False False
http_to_http_json -3.96KiB/CPU-s -0.03 68.66% 13.62MiB/CPU-s 211.14KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015135 13.62MiB/CPU-s 219.77KiB/CPU-s 2.84KiB/CPU-s 0.0 0.015758 False False
splunk_hec_route_s3 -34.64KiB/CPU-s -0.29 99.93% 11.76MiB/CPU-s 566.28KiB/CPU-s 7.3KiB/CPU-s 0.0 0.047007 11.73MiB/CPU-s 550.55KiB/CPU-s 7.1KiB/CPU-s 0.0 0.045834 False False
datadog_agent_remap_datadog_logs -104.66KiB/CPU-s -0.3 99.99% 34.02MiB/CPU-s 1.55MiB/CPU-s 20.46KiB/CPU-s 0.0 0.045531 33.92MiB/CPU-s 1.25MiB/CPU-s 16.52KiB/CPU-s 0.0 0.036865 False False
datadog_agent_remap_blackhole_acks -113.78KiB/CPU-s -0.35 100.00% 31.38MiB/CPU-s 1.03MiB/CPU-s 13.59KiB/CPU-s 0.0 0.032754 31.27MiB/CPU-s 912.71KiB/CPU-s 11.78KiB/CPU-s 0.0 0.028502 False False
syslog_log2metric_humio_metrics -24.97KiB/CPU-s -0.39 100.00% 6.2MiB/CPU-s 138.79KiB/CPU-s 1.79KiB/CPU-s 0.0 0.021845 6.18MiB/CPU-s 173.13KiB/CPU-s 2.24KiB/CPU-s 0.0 0.027358 False False
syslog_splunk_hec_logs -126.3KiB/CPU-s -1.36 100.00% 9.05MiB/CPU-s 167.77KiB/CPU-s 2.17KiB/CPU-s 0.0 0.018099 8.93MiB/CPU-s 279.85KiB/CPU-s 3.61KiB/CPU-s 0.0 0.030607 False False
datadog_agent_remap_blackhole -692.38KiB/CPU-s -2.18 100.00% 31.06MiB/CPU-s 920.75KiB/CPU-s 11.89KiB/CPU-s 0.0 0.028944 30.39MiB/CPU-s 1.39MiB/CPU-s 18.35KiB/CPU-s 0.0 0.0457 False False
syslog_regex_logs2metric_ddmetrics -96.99KiB/CPU-s -2.56 100.00% 3.7MiB/CPU-s 404.7KiB/CPU-s 5.22KiB/CPU-s 0.0 0.106894 3.6MiB/CPU-s 441.19KiB/CPU-s 5.7KiB/CPU-s 0.0 0.119596 True False
datadog_agent_remap_datadog_logs_acks -922.92KiB/CPU-s -2.61 100.00% 34.49MiB/CPU-s 999.55KiB/CPU-s 12.9KiB/CPU-s 0.0 0.028302 33.58MiB/CPU-s 927.32KiB/CPU-s 11.97KiB/CPU-s 0.0 0.026962 False False

github-actions[bot] avatar Jan 23 '23 12:01 github-actions[bot]

Regression Test Results

Run ID: 7351e79a-ea05-4bc2-8603-b51dce6e9193
Baseline: dffd4d0497837ff76b28ee1b70447720b12761e4
Comparison: 5e066038ee1f21ea8ecb430f32fdc5dc62dc983c
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_regex_logs2metric_ddmetrics 65.82KiB/CPU-s 1.77 100.00% 3.63MiB/CPU-s 442.61KiB/CPU-s 5.71KiB/CPU-s 0.0 0.119074 3.69MiB/CPU-s 423.91KiB/CPU-s 5.47KiB/CPU-s 0.0 0.11206 True False
syslog_splunk_hec_logs 76.96KiB/CPU-s 0.84 100.00% 9.0MiB/CPU-s 195.66KiB/CPU-s 2.53KiB/CPU-s 0.0 0.021235 9.07MiB/CPU-s 212.31KiB/CPU-s 2.74KiB/CPU-s 0.0 0.022852 False False
otlp_http_to_blackhole 11.34KiB/CPU-s 0.73 100.00% 1.52MiB/CPU-s 117.08KiB/CPU-s 1.51KiB/CPU-s 0.0 0.074983 1.54MiB/CPU-s 119.92KiB/CPU-s 1.55KiB/CPU-s 0.0 0.076252 False False
datadog_agent_remap_datadog_logs 227.04KiB/CPU-s 0.66 100.00% 33.76MiB/CPU-s 1.42MiB/CPU-s 18.7KiB/CPU-s 0.0 0.041937 33.98MiB/CPU-s 1.31MiB/CPU-s 17.31KiB/CPU-s 0.0 0.038569 False False
http_to_http_acks 14.29KiB/CPU-s 0.27 21.62% 5.26MiB/CPU-s 2.78MiB/CPU-s 36.79KiB/CPU-s 0.0 0.529229 5.27MiB/CPU-s 2.79MiB/CPU-s 36.89KiB/CPU-s 0.0 0.529268 True False
splunk_hec_to_splunk_hec_logs_noack 7.54KiB/CPU-s 0.05 93.60% 13.62MiB/CPU-s 251.96KiB/CPU-s 3.25KiB/CPU-s 0.0 0.018068 13.62MiB/CPU-s 189.82KiB/CPU-s 2.45KiB/CPU-s 0.0 0.013604 False False
http_to_http_noack 4.26KiB/CPU-s 0.03 55.57% 13.61MiB/CPU-s 327.93KiB/CPU-s 4.23KiB/CPU-s 0.0 0.023525 13.62MiB/CPU-s 280.02KiB/CPU-s 3.61KiB/CPU-s 0.0 0.020082 False False
otlp_grpc_to_blackhole -14.1B/CPU-s -0.0 1.32% 1.05MiB/CPU-s 34.71KiB/CPU-s 458.9B/CPU-s 0.0 0.032431 1.05MiB/CPU-s 54.27KiB/CPU-s 716.94B/CPU-s 0.0 0.050705 False False
splunk_hec_indexer_ack_blackhole 231.27B/CPU-s 0.0 3.87% 13.62MiB/CPU-s 254.69KiB/CPU-s 3.29KiB/CPU-s 0.0 0.018265 13.62MiB/CPU-s 255.36KiB/CPU-s 3.29KiB/CPU-s 0.0 0.018313 False False
file_to_blackhole 1.94KiB/CPU-s 0.0 6.96% 54.49MiB/CPU-s 1.19MiB/CPU-s 15.75KiB/CPU-s 0.0 0.021872 54.5MiB/CPU-s 1.18MiB/CPU-s 15.64KiB/CPU-s 0.0 0.021727 False False
fluent_elasticsearch -368.88B/CPU-s -0.0 49.35% 45.41MiB/CPU-s 30.06KiB/CPU-s 392.81B/CPU-s 0.0 0.000646 45.41MiB/CPU-s 30.03KiB/CPU-s 392.38B/CPU-s 0.0 0.000646 False False
http_to_http_json -123.08B/CPU-s -0.0 2.50% 13.62MiB/CPU-s 210.84KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015114 13.62MiB/CPU-s 209.47KiB/CPU-s 2.7KiB/CPU-s 0.0 0.015016 False False
splunk_hec_to_splunk_hec_logs_acks -697.15B/CPU-s -0.0 8.15% 13.62MiB/CPU-s 352.96KiB/CPU-s 4.55KiB/CPU-s 0.0 0.025313 13.62MiB/CPU-s 375.98KiB/CPU-s 4.85KiB/CPU-s 0.0 0.026965 False False
enterprise_http_to_http 271.18B/CPU-s 0.0 5.12% 13.62MiB/CPU-s 225.07KiB/CPU-s 2.91KiB/CPU-s 0.0 0.016137 13.62MiB/CPU-s 226.88KiB/CPU-s 2.93KiB/CPU-s 0.0 0.016266 False False
socket_to_socket_blackhole -35.15KiB/CPU-s -0.25 100.00% 13.5MiB/CPU-s 171.82KiB/CPU-s 2.22KiB/CPU-s 0.0 0.012424 13.47MiB/CPU-s 129.13KiB/CPU-s 1.67KiB/CPU-s 0.0 0.009361 False False
syslog_log2metric_splunk_hec_metrics -27.87KiB/CPU-s -0.29 100.00% 9.46MiB/CPU-s 256.36KiB/CPU-s 3.31KiB/CPU-s 0.0 0.026451 9.44MiB/CPU-s 264.28KiB/CPU-s 3.41KiB/CPU-s 0.0 0.027347 False False
http_text_to_http_json -139.77KiB/CPU-s -0.56 100.00% 24.3MiB/CPU-s 999.34KiB/CPU-s 12.9KiB/CPU-s 0.0 0.040158 24.16MiB/CPU-s 839.41KiB/CPU-s 10.83KiB/CPU-s 0.0 0.033922 False False
syslog_humio_logs -63.16KiB/CPU-s -0.68 100.00% 9.07MiB/CPU-s 231.44KiB/CPU-s 2.99KiB/CPU-s 0.0 0.024908 9.01MiB/CPU-s 211.25KiB/CPU-s 2.73KiB/CPU-s 0.0 0.022891 False False
syslog_loki -62.14KiB/CPU-s -0.68 100.00% 8.97MiB/CPU-s 105.8KiB/CPU-s 1.37KiB/CPU-s 0.0 0.011522 8.91MiB/CPU-s 97.43KiB/CPU-s 1.26KiB/CPU-s 0.0 0.010682 False False
splunk_hec_route_s3 -91.98KiB/CPU-s -0.76 100.00% 11.76MiB/CPU-s 579.69KiB/CPU-s 7.48KiB/CPU-s 0.0 0.04814 11.67MiB/CPU-s 584.61KiB/CPU-s 7.54KiB/CPU-s 0.0 0.048922 False False
datadog_agent_remap_datadog_logs_acks -638.05KiB/CPU-s -1.82 100.00% 34.28MiB/CPU-s 846.52KiB/CPU-s 10.93KiB/CPU-s 0.0 0.024111 33.66MiB/CPU-s 790.78KiB/CPU-s 10.21KiB/CPU-s 0.0 0.02294 False False
syslog_log2metric_humio_metrics -135.11KiB/CPU-s -2.17 100.00% 6.09MiB/CPU-s 211.3KiB/CPU-s 2.73KiB/CPU-s 0.0 0.033874 5.96MiB/CPU-s 359.92KiB/CPU-s 4.65KiB/CPU-s 0.0 0.058979 False False
datadog_agent_remap_blackhole -723.27KiB/CPU-s -2.33 100.00% 30.26MiB/CPU-s 1.79MiB/CPU-s 23.71KiB/CPU-s 0.0 0.059224 29.55MiB/CPU-s 2.1MiB/CPU-s 27.73KiB/CPU-s 0.0 0.070984 False False
datadog_agent_remap_blackhole_acks -818.24KiB/CPU-s -2.55 100.00% 31.34MiB/CPU-s 1.04MiB/CPU-s 13.71KiB/CPU-s 0.0 0.03308 30.54MiB/CPU-s 1.26MiB/CPU-s 16.59KiB/CPU-s 0.0 0.041099 False False

github-actions[bot] avatar Jan 23 '23 17:01 github-actions[bot]