vector icon indicating copy to clipboard operation
vector copied to clipboard

Published 20 hours ago • verified publisher icon vectordotdev

chore(releasing): Prepare v0.27.0 release

Open jszwedko opened this issue 2 years ago • 3 comments
trafficstars

TODOs:

  • [ ] Flesh out update for metric tag model changes
  • [ ] Flesh out update for memory instrumentation
  • [x] Readd tests for installation on older OSes
  • [ ] Description
  • [x] Cherry-pick milestone issues

jszwedko avatar Jan 12 '23 23:01 jszwedko

Regression Test Results

Run ID: 914ea1d7-7899-4f3c-9134-65f5296460d7
Baseline: 964808084ef34df3279a46069e562700a15ef842
Comparison: 154f4a6ee8e66359b3520bd038925d6422e8a157
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
datadog_agent_remap_blackhole_acks -2.24MiB/CPU-s -7.47 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_humio_metrics 148.47KiB/CPU-s 2.46 100.00% 5.9MiB/CPU-s 361.93KiB/CPU-s 4.67KiB/CPU-s 0.0 0.059911 6.04MiB/CPU-s 346.04KiB/CPU-s 4.47KiB/CPU-s 0.0 0.055906 False False
syslog_loki 129.74KiB/CPU-s 1.45 100.00% 8.75MiB/CPU-s 224.45KiB/CPU-s 2.9KiB/CPU-s 0.0 0.025044 8.88MiB/CPU-s 217.9KiB/CPU-s 2.81KiB/CPU-s 0.0 0.023967 False False
syslog_splunk_hec_logs 87.87KiB/CPU-s 0.98 100.00% 8.78MiB/CPU-s 333.57KiB/CPU-s 4.31KiB/CPU-s 0.0 0.037112 8.86MiB/CPU-s 364.08KiB/CPU-s 4.7KiB/CPU-s 0.0 0.040115 False False
syslog_humio_logs 81.05KiB/CPU-s 0.87 100.00% 9.11MiB/CPU-s 266.93KiB/CPU-s 3.44KiB/CPU-s 0.0 0.028618 9.19MiB/CPU-s 124.38KiB/CPU-s 1.61KiB/CPU-s 0.0 0.01322 False False
syslog_regex_logs2metric_ddmetrics 31.75KiB/CPU-s 0.87 100.00% 3.56MiB/CPU-s 366.04KiB/CPU-s 4.72KiB/CPU-s 0.0 0.100436 3.59MiB/CPU-s 320.16KiB/CPU-s 4.13KiB/CPU-s 0.0 0.087088 True False
datadog_agent_remap_blackhole 246.8KiB/CPU-s 0.78 100.00% 30.97MiB/CPU-s 1.16MiB/CPU-s 15.34KiB/CPU-s 0.0 0.037452 31.21MiB/CPU-s 1018.53KiB/CPU-s 13.15KiB/CPU-s 0.0 0.031867 False False
otlp_http_to_blackhole 12.08KiB/CPU-s 0.77 100.00% 1.53MiB/CPU-s 124.12KiB/CPU-s 1.6KiB/CPU-s 0.0 0.079305 1.54MiB/CPU-s 123.51KiB/CPU-s 1.59KiB/CPU-s 0.0 0.078309 False False
splunk_hec_route_s3 75.44KiB/CPU-s 0.64 100.00% 11.46MiB/CPU-s 725.64KiB/CPU-s 9.36KiB/CPU-s 0.0 0.061852 11.53MiB/CPU-s 680.44KiB/CPU-s 8.78KiB/CPU-s 0.0 0.057629 False False
http_to_http_acks 15.79KiB/CPU-s 0.29 23.90% 5.23MiB/CPU-s 2.77MiB/CPU-s 36.62KiB/CPU-s 0.0 0.529459 5.25MiB/CPU-s 2.78MiB/CPU-s 36.76KiB/CPU-s 0.0 0.529878 True False
http_to_http_json 38.45KiB/CPU-s 0.28 100.00% 13.58MiB/CPU-s 275.9KiB/CPU-s 3.56KiB/CPU-s 0.0 0.019832 13.62MiB/CPU-s 213.62KiB/CPU-s 2.76KiB/CPU-s 0.0 0.015313 False False
socket_to_socket_blackhole 18.51KiB/CPU-s 0.13 99.62% 13.52MiB/CPU-s 331.88KiB/CPU-s 4.28KiB/CPU-s 0.0 0.023974 13.54MiB/CPU-s 367.58KiB/CPU-s 4.75KiB/CPU-s 0.0 0.026517 False False
splunk_hec_to_splunk_hec_logs_noack 8.57KiB/CPU-s 0.06 96.29% 13.62MiB/CPU-s 255.23KiB/CPU-s 3.29KiB/CPU-s 0.0 0.018304 13.62MiB/CPU-s 190.77KiB/CPU-s 2.46KiB/CPU-s 0.0 0.013672 False False
http_to_http_noack 2.23KiB/CPU-s 0.02 28.05% 13.61MiB/CPU-s 353.3KiB/CPU-s 4.56KiB/CPU-s 0.0 0.025349 13.61MiB/CPU-s 328.36KiB/CPU-s 4.24KiB/CPU-s 0.0 0.023556 False False
splunk_hec_to_splunk_hec_logs_acks 1.86KiB/CPU-s 0.01 23.13% 13.62MiB/CPU-s 353.15KiB/CPU-s 4.56KiB/CPU-s 0.0 0.025327 13.62MiB/CPU-s 339.12KiB/CPU-s 4.38KiB/CPU-s 0.0 0.024318 False False
enterprise_http_to_http 3.02B/CPU-s 0.0 0.07% 13.62MiB/CPU-s 192.03KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013765 13.62MiB/CPU-s 192.52KiB/CPU-s 2.49KiB/CPU-s 0.0 0.0138 False False
fluent_elasticsearch 341.95B/CPU-s 0.0 45.99% 45.41MiB/CPU-s 30.42KiB/CPU-s 397.68B/CPU-s 0.0 0.000654 45.41MiB/CPU-s 29.96KiB/CPU-s 391.52B/CPU-s 0.0 0.000644 False False
splunk_hec_indexer_ack_blackhole -1.48KiB/CPU-s -0.01 25.42% 13.62MiB/CPU-s 247.47KiB/CPU-s 3.19KiB/CPU-s 0.0 0.017746 13.62MiB/CPU-s 254.25KiB/CPU-s 3.28KiB/CPU-s 0.0 0.018234 False False
file_to_blackhole -3.34KiB/CPU-s -0.01 12.50% 54.49MiB/CPU-s 1.09MiB/CPU-s 14.41KiB/CPU-s 0.0 0.020023 54.49MiB/CPU-s 1.18MiB/CPU-s 15.59KiB/CPU-s 0.0 0.021675 False False
datadog_agent_remap_datadog_logs -87.13KiB/CPU-s -0.25 99.99% 34.39MiB/CPU-s 1.15MiB/CPU-s 15.18KiB/CPU-s 0.0 0.033412 34.3MiB/CPU-s 1.27MiB/CPU-s 16.79KiB/CPU-s 0.0 0.037052 False False
otlp_grpc_to_blackhole -5.65KiB/CPU-s -0.53 100.00% 1.03MiB/CPU-s 46.95KiB/CPU-s 620.67B/CPU-s 0.0 0.044366 1.03MiB/CPU-s 54.9KiB/CPU-s 725.42B/CPU-s 0.0 0.05215 False False
http_text_to_http_json -146.9KiB/CPU-s -0.57 100.00% 25.22MiB/CPU-s 725.94KiB/CPU-s 9.37KiB/CPU-s 0.0 0.028102 25.08MiB/CPU-s 837.59KiB/CPU-s 10.81KiB/CPU-s 0.0 0.03261 False False
syslog_log2metric_splunk_hec_metrics -101.07KiB/CPU-s -1.04 100.00% 9.51MiB/CPU-s 136.66KiB/CPU-s 1.76KiB/CPU-s 0.0 0.014029 9.41MiB/CPU-s 204.04KiB/CPU-s 2.63KiB/CPU-s 0.0 0.021165 False False
datadog_agent_remap_datadog_logs_acks -710.66KiB/CPU-s -2.05 100.00% 33.88MiB/CPU-s 1.49MiB/CPU-s 19.69KiB/CPU-s 0.0 0.043956 33.19MiB/CPU-s 1.46MiB/CPU-s 19.27KiB/CPU-s 0.0 0.043938 False False
datadog_agent_remap_blackhole_acks -2.24MiB/CPU-s -7.47 100.00% 29.92MiB/CPU-s 1.61MiB/CPU-s 21.29KiB/CPU-s 0.0 0.05382 27.69MiB/CPU-s 8.9MiB/CPU-s 117.62KiB/CPU-s 0.0 0.321405 True False

github-actions[bot] avatar Jan 13 '23 00:01 github-actions[bot]

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 4 committers have signed the CLA.

:white_check_mark: jszwedko
:white_check_mark: dizlv
:white_check_mark: lukesteensen
:x: dependabot[bot]
You have signed the CLA already but the status is still pending? Let us recheck it.

bits-bot avatar Jan 17 '23 15:01 bits-bot

Regression Test Results

Run ID: a870b51b-c043-4dc5-bbef-07ae7f1ed50e
Baseline: 964808084ef34df3279a46069e562700a15ef842
Comparison: 0facf939ed6095332fbf0f37db0a411c9b743b69
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_regex_logs2metric_ddmetrics 116.3KiB/CPU-s 3.14 100.00% 3.62MiB/CPU-s 425.14KiB/CPU-s 5.49KiB/CPU-s 0.0 0.114768 3.73MiB/CPU-s 419.35KiB/CPU-s 5.41KiB/CPU-s 0.0 0.109758 True False
http_text_to_http_json 482.52KiB/CPU-s 1.85 100.00% 25.46MiB/CPU-s 745.66KiB/CPU-s 9.63KiB/CPU-s 0.0 0.028594 25.94MiB/CPU-s 516.14KiB/CPU-s 6.66KiB/CPU-s 0.0 0.019432 False False
datadog_agent_remap_datadog_logs 462.45KiB/CPU-s 1.34 100.00% 33.6MiB/CPU-s 1.67MiB/CPU-s 22.08KiB/CPU-s 0.0 0.049754 34.05MiB/CPU-s 1.52MiB/CPU-s 20.09KiB/CPU-s 0.0 0.044674 False False
datadog_agent_remap_blackhole_acks 314.26KiB/CPU-s 1.01 100.00% 30.26MiB/CPU-s 1.71MiB/CPU-s 22.59KiB/CPU-s 0.0 0.056464 30.56MiB/CPU-s 1.68MiB/CPU-s 22.17KiB/CPU-s 0.0 0.05485 False False
otlp_grpc_to_blackhole 10.74KiB/CPU-s 1.0 100.00% 1.04MiB/CPU-s 53.71KiB/CPU-s 709.81B/CPU-s 0.0 0.050216 1.05MiB/CPU-s 51.17KiB/CPU-s 676.01B/CPU-s 0.0 0.047369 False False
datadog_agent_remap_datadog_logs_acks 293.97KiB/CPU-s 0.85 100.00% 33.97MiB/CPU-s 1.16MiB/CPU-s 15.38KiB/CPU-s 0.0 0.034256 34.26MiB/CPU-s 1.14MiB/CPU-s 15.1KiB/CPU-s 0.0 0.033344 False False
otlp_http_to_blackhole 12.05KiB/CPU-s 0.78 100.00% 1.52MiB/CPU-s 120.14KiB/CPU-s 1.55KiB/CPU-s 0.0 0.077273 1.53MiB/CPU-s 118.38KiB/CPU-s 1.53KiB/CPU-s 0.0 0.075558 False False
datadog_agent_remap_blackhole 201.67KiB/CPU-s 0.63 100.00% 31.17MiB/CPU-s 1.25MiB/CPU-s 16.48KiB/CPU-s 0.0 0.039996 31.37MiB/CPU-s 706.46KiB/CPU-s 9.12KiB/CPU-s 0.0 0.021992 False False
enterprise_http_to_http 14.76KiB/CPU-s 0.11 99.32% 13.61MiB/CPU-s 375.54KiB/CPU-s 4.84KiB/CPU-s 0.0 0.026949 13.62MiB/CPU-s 192.94KiB/CPU-s 2.49KiB/CPU-s 0.0 0.013831 False False
http_to_http_acks 2.91KiB/CPU-s 0.05 4.48% 5.24MiB/CPU-s 2.72MiB/CPU-s 36.01KiB/CPU-s 0.0 0.519475 5.25MiB/CPU-s 2.81MiB/CPU-s 37.14KiB/CPU-s 0.0 0.535615 True False
splunk_hec_to_splunk_hec_logs_noack 3.35KiB/CPU-s 0.02 62.65% 13.62MiB/CPU-s 219.21KiB/CPU-s 2.83KiB/CPU-s 0.0 0.015714 13.63MiB/CPU-s 192.24KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013777 False False
syslog_log2metric_humio_metrics 985.58B/CPU-s 0.02 12.71% 5.93MiB/CPU-s 369.91KiB/CPU-s 4.78KiB/CPU-s 0.0 0.060906 5.93MiB/CPU-s 283.42KiB/CPU-s 3.66KiB/CPU-s 0.0 0.046658 False False
http_to_http_noack 2.03KiB/CPU-s 0.01 31.94% 13.62MiB/CPU-s 282.19KiB/CPU-s 3.64KiB/CPU-s 0.0 0.020238 13.62MiB/CPU-s 257.36KiB/CPU-s 3.32KiB/CPU-s 0.0 0.018455 False False
fluent_elasticsearch -326.68B/CPU-s -0.0 44.30% 45.41MiB/CPU-s 30.21KiB/CPU-s 394.97B/CPU-s 0.0 0.00065 45.41MiB/CPU-s 29.95KiB/CPU-s 391.6B/CPU-s 0.0 0.000644 False False
file_to_blackhole -2.49KiB/CPU-s -0.0 9.63% 54.5MiB/CPU-s 1.1MiB/CPU-s 14.53KiB/CPU-s 0.0 0.02018 54.5MiB/CPU-s 1.1MiB/CPU-s 14.58KiB/CPU-s 0.0 0.020265 False False
splunk_hec_to_splunk_hec_logs_acks -336.82B/CPU-s -0.0 4.08% 13.61MiB/CPU-s 352.03KiB/CPU-s 4.54KiB/CPU-s 0.0 0.025249 13.61MiB/CPU-s 353.68KiB/CPU-s 4.56KiB/CPU-s 0.0 0.025368 False False
splunk_hec_indexer_ack_blackhole -2.79KiB/CPU-s -0.02 45.92% 13.62MiB/CPU-s 243.11KiB/CPU-s 3.14KiB/CPU-s 0.0 0.017431 13.62MiB/CPU-s 256.14KiB/CPU-s 3.3KiB/CPU-s 0.0 0.018368 False False
socket_to_socket_blackhole -6.1KiB/CPU-s -0.04 54.79% 13.47MiB/CPU-s 468.13KiB/CPU-s 6.04KiB/CPU-s 0.0 0.033948 13.46MiB/CPU-s 419.01KiB/CPU-s 5.41KiB/CPU-s 0.0 0.030399 False False
splunk_hec_route_s3 -12.36KiB/CPU-s -0.1 74.42% 11.83MiB/CPU-s 543.84KiB/CPU-s 7.02KiB/CPU-s 0.0 0.04489 11.82MiB/CPU-s 643.77KiB/CPU-s 8.31KiB/CPU-s 0.0 0.053193 False False
http_to_http_json -54.28KiB/CPU-s -0.39 100.00% 13.61MiB/CPU-s 257.41KiB/CPU-s 3.32KiB/CPU-s 0.0 0.018472 13.55MiB/CPU-s 321.78KiB/CPU-s 4.15KiB/CPU-s 0.0 0.023183 False False
syslog_loki -106.83KiB/CPU-s -1.17 100.00% 8.95MiB/CPU-s 173.31KiB/CPU-s 2.24KiB/CPU-s 0.0 0.018915 8.84MiB/CPU-s 119.95KiB/CPU-s 1.55KiB/CPU-s 0.0 0.013246 False False
syslog_log2metric_splunk_hec_metrics -123.69KiB/CPU-s -1.28 100.00% 9.41MiB/CPU-s 375.91KiB/CPU-s 4.85KiB/CPU-s 0.0 0.039004 9.29MiB/CPU-s 329.18KiB/CPU-s 4.25KiB/CPU-s 0.0 0.034599 False False
syslog_splunk_hec_logs -141.09KiB/CPU-s -1.51 100.00% 9.11MiB/CPU-s 81.64KiB/CPU-s 1.05KiB/CPU-s 0.0 0.008755 8.97MiB/CPU-s 180.87KiB/CPU-s 2.33KiB/CPU-s 0.0 0.019694 False False
syslog_humio_logs -169.78KiB/CPU-s -1.82 100.00% 9.11MiB/CPU-s 338.47KiB/CPU-s 4.37KiB/CPU-s 0.0 0.036296 8.94MiB/CPU-s 275.78KiB/CPU-s 3.56KiB/CPU-s 0.0 0.030122 False False

github-actions[bot] avatar Jan 17 '23 16:01 github-actions[bot]

Regression Test Results

Run ID: c9d61285-c008-4321-93c4-42bdc300973f
Baseline: 964808084ef34df3279a46069e562700a15ef842
Comparison: e150c49f9c99ab7af2ef590356d15892fe544825
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
syslog_log2metric_humio_metrics -625.06KiB/CPU-s -9.79 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_datadog_logs 1.04MiB/CPU-s 3.1 100.00% 33.62MiB/CPU-s 1.89MiB/CPU-s 24.97KiB/CPU-s 0.0 0.056221 34.66MiB/CPU-s 1.21MiB/CPU-s 15.95KiB/CPU-s 0.0 0.034831 False False
syslog_regex_logs2metric_ddmetrics 86.99KiB/CPU-s 2.38 100.00% 3.57MiB/CPU-s 504.58KiB/CPU-s 6.51KiB/CPU-s 0.0 0.138062 3.65MiB/CPU-s 486.6KiB/CPU-s 6.28KiB/CPU-s 0.0 0.130045 True False
datadog_agent_remap_blackhole_acks 666.76KiB/CPU-s 2.12 100.00% 30.69MiB/CPU-s 1.93MiB/CPU-s 25.57KiB/CPU-s 0.0 0.062996 31.34MiB/CPU-s 1.22MiB/CPU-s 16.08KiB/CPU-s 0.0 0.038807 False False
datadog_agent_remap_datadog_logs_acks 543.22KiB/CPU-s 1.57 100.00% 33.7MiB/CPU-s 1.53MiB/CPU-s 20.22KiB/CPU-s 0.0 0.045403 34.23MiB/CPU-s 1.74MiB/CPU-s 22.96KiB/CPU-s 0.0 0.050761 False False
splunk_hec_route_s3 121.81KiB/CPU-s 1.0 100.00% 11.86MiB/CPU-s 535.94KiB/CPU-s 6.91KiB/CPU-s 0.0 0.044124 11.98MiB/CPU-s 507.72KiB/CPU-s 6.55KiB/CPU-s 0.0 0.041385 False False
socket_to_socket_blackhole 88.61KiB/CPU-s 0.63 100.00% 13.65MiB/CPU-s 342.45KiB/CPU-s 4.42KiB/CPU-s 0.0 0.024505 13.73MiB/CPU-s 133.79KiB/CPU-s 1.73KiB/CPU-s 0.0 0.009514 False False
otlp_http_to_blackhole 9.72KiB/CPU-s 0.62 100.00% 1.52MiB/CPU-s 116.23KiB/CPU-s 1.5KiB/CPU-s 0.0 0.074553 1.53MiB/CPU-s 120.93KiB/CPU-s 1.56KiB/CPU-s 0.0 0.07709 False False
http_text_to_http_json 74.85KiB/CPU-s 0.29 100.00% 25.23MiB/CPU-s 754.5KiB/CPU-s 9.74KiB/CPU-s 0.0 0.029198 25.31MiB/CPU-s 1.01MiB/CPU-s 13.28KiB/CPU-s 0.0 0.039719 False False
syslog_loki 12.81KiB/CPU-s 0.14 99.57% 8.64MiB/CPU-s 285.19KiB/CPU-s 3.68KiB/CPU-s 0.0 0.032238 8.65MiB/CPU-s 198.12KiB/CPU-s 2.56KiB/CPU-s 0.0 0.022362 False False
syslog_log2metric_splunk_hec_metrics 11.71KiB/CPU-s 0.12 87.73% 9.16MiB/CPU-s 435.77KiB/CPU-s 5.63KiB/CPU-s 0.0 0.046476 9.17MiB/CPU-s 393.63KiB/CPU-s 5.08KiB/CPU-s 0.0 0.04193 False False
enterprise_http_to_http 12.35KiB/CPU-s 0.09 99.19% 13.61MiB/CPU-s 329.21KiB/CPU-s 4.25KiB/CPU-s 0.0 0.023616 13.62MiB/CPU-s 149.78KiB/CPU-s 1.93KiB/CPU-s 0.0 0.010735 False False
splunk_hec_to_splunk_hec_logs_noack 8.05KiB/CPU-s 0.06 95.28% 13.62MiB/CPU-s 250.57KiB/CPU-s 3.23KiB/CPU-s 0.0 0.017968 13.63MiB/CPU-s 190.13KiB/CPU-s 2.45KiB/CPU-s 0.0 0.013626 False False
fluent_elasticsearch 32.94B/CPU-s 0.0 4.71% 45.41MiB/CPU-s 30.64KiB/CPU-s 400.59B/CPU-s 0.0 0.000659 45.41MiB/CPU-s 29.72KiB/CPU-s 388.43B/CPU-s 0.0 0.000639 False False
file_to_blackhole 233.37B/CPU-s 0.0 0.86% 54.49MiB/CPU-s 1.1MiB/CPU-s 14.49KiB/CPU-s 0.0 0.020131 54.49MiB/CPU-s 1.17MiB/CPU-s 15.5KiB/CPU-s 0.0 0.02155 False False
splunk_hec_indexer_ack_blackhole -1.76KiB/CPU-s -0.01 30.64% 13.62MiB/CPU-s 238.91KiB/CPU-s 3.08KiB/CPU-s 0.0 0.01713 13.62MiB/CPU-s 251.62KiB/CPU-s 3.25KiB/CPU-s 0.0 0.018043 False False
splunk_hec_to_splunk_hec_logs_acks -3.46KiB/CPU-s -0.02 39.00% 13.62MiB/CPU-s 367.12KiB/CPU-s 4.74KiB/CPU-s 0.0 0.026327 13.61MiB/CPU-s 375.43KiB/CPU-s 4.84KiB/CPU-s 0.0 0.02693 False False
http_to_http_noack -4.08KiB/CPU-s -0.03 56.83% 13.62MiB/CPU-s 258.72KiB/CPU-s 3.34KiB/CPU-s 0.0 0.018552 13.61MiB/CPU-s 307.9KiB/CPU-s 3.97KiB/CPU-s 0.0 0.022085 False False
http_to_http_acks -2.67KiB/CPU-s -0.05 4.17% 5.23MiB/CPU-s 2.75MiB/CPU-s 36.29KiB/CPU-s 0.0 0.524997 5.23MiB/CPU-s 2.73MiB/CPU-s 36.05KiB/CPU-s 0.0 0.521824 True False
http_to_http_json -47.01KiB/CPU-s -0.34 100.00% 13.53MiB/CPU-s 333.9KiB/CPU-s 4.31KiB/CPU-s 0.0 0.0241 13.48MiB/CPU-s 392.57KiB/CPU-s 5.06KiB/CPU-s 0.0 0.02843 False False
otlp_grpc_to_blackhole -9.94KiB/CPU-s -0.92 100.00% 1.05MiB/CPU-s 39.96KiB/CPU-s 528.18B/CPU-s 0.0 0.037008 1.04MiB/CPU-s 56.38KiB/CPU-s 744.8B/CPU-s 0.0 0.052701 False False
syslog_splunk_hec_logs -86.26KiB/CPU-s -0.94 100.00% 8.92MiB/CPU-s 298.06KiB/CPU-s 3.85KiB/CPU-s 0.0 0.032643 8.83MiB/CPU-s 342.24KiB/CPU-s 4.42KiB/CPU-s 0.0 0.03784 False False
datadog_agent_remap_blackhole -444.57KiB/CPU-s -1.37 100.00% 31.62MiB/CPU-s 748.84KiB/CPU-s 9.67KiB/CPU-s 0.0 0.023124 31.19MiB/CPU-s 1.22MiB/CPU-s 16.12KiB/CPU-s 0.0 0.039115 False False
syslog_humio_logs -322.71KiB/CPU-s -3.58 100.00% 8.81MiB/CPU-s 489.14KiB/CPU-s 6.31KiB/CPU-s 0.0 0.054196 8.5MiB/CPU-s 378.01KiB/CPU-s 4.88KiB/CPU-s 0.0 0.043436 False False
syslog_log2metric_humio_metrics -625.06KiB/CPU-s -9.79 100.00% 6.23MiB/CPU-s 167.36KiB/CPU-s 2.16KiB/CPU-s 0.0 0.026223 5.62MiB/CPU-s 581.47KiB/CPU-s 7.5KiB/CPU-s 0.0 0.100999 True False

github-actions[bot] avatar Jan 17 '23 19:01 github-actions[bot]

Regression Test Results

Run ID: 4d0167a0-8abf-4f36-b6dd-8558055a7402
Baseline: 964808084ef34df3279a46069e562700a15ef842
Comparison: a2ec1a496a1bb7a3c4cefd5662150820c1b8ab10
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
syslog_regex_logs2metric_ddmetrics -191.68KiB/CPU-s -5.15 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json 404.91KiB/CPU-s 1.55 100.00% 25.44MiB/CPU-s 638.59KiB/CPU-s 8.24KiB/CPU-s 0.0 0.024512 25.83MiB/CPU-s 598.92KiB/CPU-s 7.73KiB/CPU-s 0.0 0.022638 False False
datadog_agent_remap_datadog_logs_acks 423.31KiB/CPU-s 1.25 100.00% 33.08MiB/CPU-s 2.11MiB/CPU-s 27.92KiB/CPU-s 0.0 0.063884 33.49MiB/CPU-s 1.62MiB/CPU-s 21.44KiB/CPU-s 0.0 0.048439 False False
otlp_http_to_blackhole 18.9KiB/CPU-s 1.21 100.00% 1.53MiB/CPU-s 109.17KiB/CPU-s 1.41KiB/CPU-s 0.0 0.069735 1.55MiB/CPU-s 115.02KiB/CPU-s 1.48KiB/CPU-s 0.0 0.072595 False False
splunk_hec_route_s3 137.48KiB/CPU-s 1.15 100.00% 11.64MiB/CPU-s 588.17KiB/CPU-s 7.59KiB/CPU-s 0.0 0.049336 11.78MiB/CPU-s 660.06KiB/CPU-s 8.52KiB/CPU-s 0.0 0.054734 False False
http_to_http_acks 27.51KiB/CPU-s 0.51 40.98% 5.24MiB/CPU-s 2.71MiB/CPU-s 35.86KiB/CPU-s 0.0 0.517785 5.27MiB/CPU-s 2.75MiB/CPU-s 36.38KiB/CPU-s 0.0 0.522525 True False
enterprise_http_to_http 6.58KiB/CPU-s 0.05 82.39% 13.61MiB/CPU-s 311.52KiB/CPU-s 4.02KiB/CPU-s 0.0 0.022345 13.62MiB/CPU-s 212.15KiB/CPU-s 2.74KiB/CPU-s 0.0 0.015209 False False
splunk_hec_to_splunk_hec_logs_noack 5.87KiB/CPU-s 0.04 86.57% 13.62MiB/CPU-s 237.38KiB/CPU-s 3.06KiB/CPU-s 0.0 0.01702 13.63MiB/CPU-s 189.35KiB/CPU-s 2.44KiB/CPU-s 0.0 0.013571 False False
splunk_hec_indexer_ack_blackhole 1003.53B/CPU-s 0.01 17.12% 13.62MiB/CPU-s 251.5KiB/CPU-s 3.24KiB/CPU-s 0.0 0.018035 13.62MiB/CPU-s 245.13KiB/CPU-s 3.16KiB/CPU-s 0.0 0.017576 False False
splunk_hec_to_splunk_hec_logs_acks 399.86B/CPU-s 0.0 4.80% 13.62MiB/CPU-s 360.47KiB/CPU-s 4.65KiB/CPU-s 0.0 0.025853 13.62MiB/CPU-s 350.75KiB/CPU-s 4.53KiB/CPU-s 0.0 0.025155 False False
fluent_elasticsearch -234.9B/CPU-s -0.0 32.60% 45.41MiB/CPU-s 30.09KiB/CPU-s 393.33B/CPU-s 0.0 0.000647 45.41MiB/CPU-s 30.34KiB/CPU-s 396.43B/CPU-s 0.0 0.000652 False False
file_to_blackhole -10.49KiB/CPU-s -0.02 37.58% 54.5MiB/CPU-s 1.08MiB/CPU-s 14.33KiB/CPU-s 0.0 0.019902 54.49MiB/CPU-s 1.21MiB/CPU-s 15.92KiB/CPU-s 0.0 0.022136 False False
http_to_http_noack -4.26KiB/CPU-s -0.03 58.61% 13.62MiB/CPU-s 258.98KiB/CPU-s 3.34KiB/CPU-s 0.0 0.01857 13.61MiB/CPU-s 310.01KiB/CPU-s 4.0KiB/CPU-s 0.0 0.022236 False False
syslog_loki -12.52KiB/CPU-s -0.14 97.88% 8.78MiB/CPU-s 381.43KiB/CPU-s 4.92KiB/CPU-s 0.0 0.042436 8.76MiB/CPU-s 178.44KiB/CPU-s 2.3KiB/CPU-s 0.0 0.01988 False False
http_to_http_json -28.22KiB/CPU-s -0.2 100.00% 13.6MiB/CPU-s 246.27KiB/CPU-s 3.18KiB/CPU-s 0.0 0.017677 13.58MiB/CPU-s 281.5KiB/CPU-s 3.63KiB/CPU-s 0.0 0.020247 False False
socket_to_socket_blackhole -26.91KiB/CPU-s -0.2 99.13% 13.25MiB/CPU-s 595.9KiB/CPU-s 7.69KiB/CPU-s 0.0 0.043908 13.23MiB/CPU-s 526.32KiB/CPU-s 6.8KiB/CPU-s 0.0 0.038858 False False
otlp_grpc_to_blackhole -3.83KiB/CPU-s -0.36 99.98% 1.03MiB/CPU-s 50.54KiB/CPU-s 668.0B/CPU-s 0.0 0.04792 1.03MiB/CPU-s 61.24KiB/CPU-s 809.16B/CPU-s 0.0 0.058282 False False
datadog_agent_remap_blackhole -191.12KiB/CPU-s -0.6 100.00% 31.14MiB/CPU-s 771.95KiB/CPU-s 9.97KiB/CPU-s 0.0 0.024205 30.96MiB/CPU-s 897.47KiB/CPU-s 11.58KiB/CPU-s 0.0 0.02831 False False
datadog_agent_remap_datadog_logs -483.21KiB/CPU-s -1.4 100.00% 33.8MiB/CPU-s 1.45MiB/CPU-s 19.21KiB/CPU-s 0.0 0.04302 33.33MiB/CPU-s 2.03MiB/CPU-s 26.84KiB/CPU-s 0.0 0.060955 False False
syslog_log2metric_splunk_hec_metrics -197.2KiB/CPU-s -2.02 100.00% 9.52MiB/CPU-s 101.34KiB/CPU-s 1.31KiB/CPU-s 0.0 0.010391 9.33MiB/CPU-s 222.07KiB/CPU-s 2.87KiB/CPU-s 0.0 0.023241 False False
syslog_humio_logs -208.82KiB/CPU-s -2.25 100.00% 9.04MiB/CPU-s 193.99KiB/CPU-s 2.5KiB/CPU-s 0.0 0.020946 8.84MiB/CPU-s 199.89KiB/CPU-s 2.58KiB/CPU-s 0.0 0.022082 False False
datadog_agent_remap_blackhole_acks -921.23KiB/CPU-s -2.87 100.00% 31.3MiB/CPU-s 958.51KiB/CPU-s 12.38KiB/CPU-s 0.0 0.029899 30.4MiB/CPU-s 1.13MiB/CPU-s 15.0KiB/CPU-s 0.0 0.037326 False False
syslog_log2metric_humio_metrics -223.75KiB/CPU-s -3.51 100.00% 6.22MiB/CPU-s 119.69KiB/CPU-s 1.55KiB/CPU-s 0.0 0.018785 6.0MiB/CPU-s 265.09KiB/CPU-s 3.42KiB/CPU-s 0.0 0.043121 False False
syslog_splunk_hec_logs -331.47KiB/CPU-s -3.6 100.00% 8.99MiB/CPU-s 141.42KiB/CPU-s 1.83KiB/CPU-s 0.0 0.015368 8.66MiB/CPU-s 308.77KiB/CPU-s 3.98KiB/CPU-s 0.0 0.034807 False False
syslog_regex_logs2metric_ddmetrics -191.68KiB/CPU-s -5.15 100.00% 3.63MiB/CPU-s 422.01KiB/CPU-s 5.45KiB/CPU-s 0.0 0.113477 3.44MiB/CPU-s 492.27KiB/CPU-s 6.36KiB/CPU-s 0.0 0.139562 True False

github-actions[bot] avatar Jan 18 '23 15:01 github-actions[bot]

Regression Test Results

Run ID: 7b3ae8d1-c156-40b6-be49-214e87061c42
Baseline: 5623d1edd4e7be4cab1651432b96f6a82c0a9c34
Comparison: 5623d1edd4e7be4cab1651432b96f6a82c0a9c34
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
datadog_agent_remap_blackhole_acks 1.77MiB/CPU-s 5.95 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole_acks 1.77MiB/CPU-s 5.95 100.00% 29.74MiB/CPU-s 2.77MiB/CPU-s 36.63KiB/CPU-s 0.0 0.093152 31.51MiB/CPU-s 793.64KiB/CPU-s 10.25KiB/CPU-s 0.0 0.024597 False False
syslog_humio_logs 206.57KiB/CPU-s 2.28 100.00% 8.83MiB/CPU-s 383.83KiB/CPU-s 4.95KiB/CPU-s 0.0 0.042453 9.03MiB/CPU-s 109.77KiB/CPU-s 1.42KiB/CPU-s 0.0 0.01187 False False
syslog_log2metric_splunk_hec_metrics 151.64KiB/CPU-s 1.62 100.00% 9.15MiB/CPU-s 500.73KiB/CPU-s 6.46KiB/CPU-s 0.0 0.053428 9.3MiB/CPU-s 234.38KiB/CPU-s 3.03KiB/CPU-s 0.0 0.02461 False False
splunk_hec_route_s3 169.84KiB/CPU-s 1.43 100.00% 11.63MiB/CPU-s 549.82KiB/CPU-s 7.09KiB/CPU-s 0.0 0.046164 11.8MiB/CPU-s 539.52KiB/CPU-s 6.97KiB/CPU-s 0.0 0.044662 False False
http_to_http_json 41.06KiB/CPU-s 0.3 100.00% 13.56MiB/CPU-s 314.82KiB/CPU-s 4.06KiB/CPU-s 0.0 0.022666 13.6MiB/CPU-s 253.77KiB/CPU-s 3.27KiB/CPU-s 0.0 0.018217 False False
datadog_agent_remap_datadog_logs 66.5KiB/CPU-s 0.19 99.76% 34.18MiB/CPU-s 1.18MiB/CPU-s 15.61KiB/CPU-s 0.0 0.034559 34.25MiB/CPU-s 1.16MiB/CPU-s 15.32KiB/CPU-s 0.0 0.033872 False False
http_to_http_acks 10.18KiB/CPU-s 0.19 15.85% 5.21MiB/CPU-s 2.76MiB/CPU-s 36.46KiB/CPU-s 0.0 0.529815 5.22MiB/CPU-s 2.69MiB/CPU-s 35.56KiB/CPU-s 0.0 0.515745 True False
socket_to_socket_blackhole 25.01KiB/CPU-s 0.18 100.00% 13.66MiB/CPU-s 195.97KiB/CPU-s 2.53KiB/CPU-s 0.0 0.014004 13.69MiB/CPU-s 157.03KiB/CPU-s 2.03KiB/CPU-s 0.0 0.011201 False False
datadog_agent_remap_blackhole 55.43KiB/CPU-s 0.17 99.95% 31.34MiB/CPU-s 983.74KiB/CPU-s 12.7KiB/CPU-s 0.0 0.030648 31.4MiB/CPU-s 744.79KiB/CPU-s 9.61KiB/CPU-s 0.0 0.023164 False False
enterprise_http_to_http 8.21KiB/CPU-s 0.06 88.65% 13.61MiB/CPU-s 332.53KiB/CPU-s 4.29KiB/CPU-s 0.0 0.023855 13.62MiB/CPU-s 225.48KiB/CPU-s 2.91KiB/CPU-s 0.0 0.016165 False False
splunk_hec_to_splunk_hec_logs_noack 5.9KiB/CPU-s 0.04 86.43% 13.62MiB/CPU-s 238.64KiB/CPU-s 3.08KiB/CPU-s 0.0 0.017109 13.63MiB/CPU-s 192.41KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013789 False False
http_to_http_noack 6.16KiB/CPU-s 0.04 74.47% 13.61MiB/CPU-s 330.72KiB/CPU-s 4.27KiB/CPU-s 0.0 0.023725 13.62MiB/CPU-s 258.5KiB/CPU-s 3.34KiB/CPU-s 0.0 0.018536 False False
otlp_grpc_to_blackhole 41.46B/CPU-s 0.0 3.94% 1.05MiB/CPU-s 36.53KiB/CPU-s 482.93B/CPU-s 0.0 0.034097 1.05MiB/CPU-s 52.04KiB/CPU-s 687.49B/CPU-s 0.0 0.04857 False False
fluent_elasticsearch 327.77B/CPU-s 0.0 44.66% 45.41MiB/CPU-s 29.66KiB/CPU-s 387.79B/CPU-s 0.0 0.000638 45.41MiB/CPU-s 30.15KiB/CPU-s 394.25B/CPU-s 0.0 0.000648 False False
splunk_hec_to_splunk_hec_logs_acks 387.9B/CPU-s 0.0 4.77% 13.62MiB/CPU-s 346.71KiB/CPU-s 4.47KiB/CPU-s 0.0 0.024865 13.62MiB/CPU-s 347.29KiB/CPU-s 4.48KiB/CPU-s 0.0 0.024906 False False
splunk_hec_indexer_ack_blackhole -2.02KiB/CPU-s -0.01 33.91% 13.62MiB/CPU-s 246.31KiB/CPU-s 3.18KiB/CPU-s 0.0 0.017661 13.62MiB/CPU-s 257.3KiB/CPU-s 3.32KiB/CPU-s 0.0 0.018451 False False
file_to_blackhole -17.53KiB/CPU-s -0.03 58.05% 54.5MiB/CPU-s 1.09MiB/CPU-s 14.4KiB/CPU-s 0.0 0.019999 54.48MiB/CPU-s 1.23MiB/CPU-s 16.25KiB/CPU-s 0.0 0.022602 False False
http_text_to_http_json -20.51KiB/CPU-s -0.08 90.06% 25.74MiB/CPU-s 723.44KiB/CPU-s 9.34KiB/CPU-s 0.0 0.027444 25.72MiB/CPU-s 637.3KiB/CPU-s 8.22KiB/CPU-s 0.0 0.024195 False False
syslog_log2metric_humio_metrics -5.85KiB/CPU-s -0.1 69.63% 6.01MiB/CPU-s 307.91KiB/CPU-s 3.98KiB/CPU-s 0.0 0.050043 6.0MiB/CPU-s 314.8KiB/CPU-s 4.06KiB/CPU-s 0.0 0.051211 False False
datadog_agent_remap_datadog_logs_acks -72.61KiB/CPU-s -0.21 99.99% 34.42MiB/CPU-s 1.14MiB/CPU-s 15.11KiB/CPU-s 0.0 0.033227 34.35MiB/CPU-s 834.05KiB/CPU-s 10.76KiB/CPU-s 0.0 0.023713 False False
otlp_http_to_blackhole -3.86KiB/CPU-s -0.24 95.37% 1.56MiB/CPU-s 99.66KiB/CPU-s 1.29KiB/CPU-s 0.0 0.062321 1.56MiB/CPU-s 112.04KiB/CPU-s 1.45KiB/CPU-s 0.0 0.070228 False False
syslog_loki -26.67KiB/CPU-s -0.3 100.00% 8.8MiB/CPU-s 198.52KiB/CPU-s 2.56KiB/CPU-s 0.0 0.022021 8.78MiB/CPU-s 145.46KiB/CPU-s 1.88KiB/CPU-s 0.0 0.016183 False False
syslog_regex_logs2metric_ddmetrics -23.91KiB/CPU-s -0.63 99.79% 3.71MiB/CPU-s 438.12KiB/CPU-s 5.66KiB/CPU-s 0.0 0.11528 3.69MiB/CPU-s 414.37KiB/CPU-s 5.35KiB/CPU-s 0.0 0.109722 True False
syslog_splunk_hec_logs -90.19KiB/CPU-s -0.98 100.00% 8.96MiB/CPU-s 213.41KiB/CPU-s 2.76KiB/CPU-s 0.0 0.023266 8.87MiB/CPU-s 335.31KiB/CPU-s 4.33KiB/CPU-s 0.0 0.03692 False False

github-actions[bot] avatar Jan 18 '23 15:01 github-actions[bot]