vector icon indicating copy to clipboard operation
vector copied to clipboard

Published 20 hours ago • verified publisher icon vectordotdev

chore(deps): bump criterion from 0.3.6 to 0.4.0

Open dependabot[bot] opened this issue 3 years ago • 5 comments
trafficstars

Bumps criterion from 0.3.6 to 0.4.0.

Changelog

Sourced from criterion's changelog.

[0.4.0] - 2022-09-10

Removed

  • The Criterion::can_plot function has been removed.
  • The Criterion::bench_function_over_inputs function has been removed.
  • The Criterion::bench_functions function has been removed.
  • The Criterion::bench function has been removed.

Changed

  • HTML report hidden behind non-default feature flag: 'html_reports'
  • Standalone support (ie without cargo-criterion) feature flag: 'cargo_bench_support'
  • MSRV bumped to 1.57
  • rayon and plotters are optional (and default) dependencies.
  • Status messages ('warming up', 'analyzing', etc) are printed to stderr, benchmark results are printed to stdout.
  • Accept subsecond durations for --warm-up-time, --measurement-time and --profile-time.
  • Replaced serde_cbor with ciborium because the former is no longer maintained.
  • Upgrade clap to v3 and regex to v1.5.

Added

  • A --discard-baseline flag for discarding rather than saving benchmark results.
  • Formal support for benchmarking code compiled to web-assembly.
  • A --quiet flag for printing just a single line per benchmark.
  • A Throughput::BytesDecimal option for measuring throughput in bytes but printing them using decimal units like kilobytes instead of binary units like kibibytes.

Fixed

  • When using bench_with_input, the input parameter will now be passed through black_box before passing it to the benchmark.
Commits
  • 5e27b69 Merge branch 'version-0.4'
  • 4d6d69a Increment version numbers.
  • 935c632 Add Throughput::BytesDecimal. Fixes #581.
  • f82ce59 Remove critcmp code (it belongs in cargo-criterion) (#610)
  • a18d080 Merge branch 'master' into version-0.4
  • f9c6b8d Merge pull request #608 from Cryptex-github/patch-1
  • 8d0224e Fix html report path
  • 2934163 Add missing black_box for bench_with_input parameters. Fixes 566.
  • dfd7b65 Add duplicated benchmark ID to assertion message.
  • ce8259e Bump criterion-plot version number.
  • Additional commits viewable in compare view

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependabot[bot] avatar Sep 12 '22 04:09 dependabot[bot]

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

bits-bot avatar Sep 12 '22 04:09 bits-bot

Deploy Preview for vector-project ready!

Name Link
Latest commit 49338ee89de810fa3e94d8abfe246d222f12a6c4
Latest deploy log https://app.netlify.com/sites/vector-project/deploys/6329b343ea5cff0009a3373f
Deploy Preview https://deploy-preview-14372--vector-project.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify[bot] avatar Sep 12 '22 04:09 netlify[bot]

Soak Test Results

Baseline: ad2b3adf0d59bd546234839f99f1a80a4323ad02 Comparison: c3366a1190aa31e16b18ac3dcbdd99b77bcd96d4 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_no_grok_blackhole 113.56KiB 1.08 100.00% 10.22MiB 150.41KiB 3.07KiB 0 0.0143654 10.33MiB 979.02KiB 19.92KiB 0 0.0924997 False False
http_pipelines_blackhole_acks 13.33KiB 1.06 100.00% 1.22MiB 106.06KiB 2.16KiB 0 0.0846031 1.24MiB 68.31KiB 1.39KiB 0 0.0539186 False False
splunk_hec_route_s3 71.78KiB 0.38 72.73% 18.57MiB 2.24MiB 46.6KiB 0 0.12044 18.64MiB 2.2MiB 45.94KiB 0 0.117828 False False
http_to_http_acks 66.66KiB 0.37 21.94% 17.4MiB 8.06MiB 168.42KiB 0 0.46283 17.47MiB 8.15MiB 169.96KiB 0 0.466305 True True
socket_to_socket_blackhole 27.55KiB 0.12 86.22% 22.55MiB 680.06KiB 13.88KiB 0 0.0294424 22.58MiB 603.39KiB 12.32KiB 0 0.0260919 False False
syslog_humio_logs 10.85KiB 0.07 95.18% 15.54MiB 200.52KiB 4.09KiB 0 0.0126013 15.55MiB 178.55KiB 3.66KiB 0 0.0112134 False False
syslog_regex_logs2metric_ddmetrics 7.15KiB 0.06 33.74% 12.34MiB 631.17KiB 12.85KiB 0 0.0499351 12.35MiB 497.9KiB 10.15KiB 0 0.0393688 False False
splunk_hec_to_splunk_hec_logs_noack 5.71KiB 0.02 41.47% 23.83MiB 384.67KiB 7.86KiB 0 0.0157586 23.84MiB 338.8KiB 6.92KiB 0 0.0138763 False False
enterprise_http_to_http -224.79B -0 2.38% 23.85MiB 254.8KiB 5.2KiB 0 0.0104327 23.85MiB 255.06KiB 5.22KiB 0 0.0104436 False False
datadog_agent_remap_blackhole 530.22B 0 0.31% 59.35MiB 4.74MiB 98.77KiB 0 0.0797942 59.35MiB 4.37MiB 91.32KiB 0 0.0736832 False False
syslog_splunk_hec_logs -5.71KiB -0.04 25.08% 15.9MiB 715.84KiB 14.57KiB 0 0.0439439 15.9MiB 506.67KiB 10.34KiB 0 0.0311145 False False
splunk_hec_indexer_ack_blackhole -10.23KiB -0.04 30.72% 23.76MiB 889.1KiB 18.09KiB 0 0.0365403 23.75MiB 910.41KiB 18.52KiB 0 0.0374317 False False
splunk_hec_to_splunk_hec_logs_acks -10.21KiB -0.04 32.99% 23.76MiB 814.98KiB 16.58KiB 0 0.0334865 23.75MiB 850.5KiB 17.3KiB 0 0.0349607 False False
file_to_blackhole -61.84KiB -0.06 42.63% 95.34MiB 3.58MiB 74.29KiB 0 0.037582 95.28MiB 3.9MiB 80.98KiB 0 0.0408815 False False
http_text_to_http_json -47.36KiB -0.12 96.27% 39.16MiB 791.14KiB 16.15KiB 0 0.0197232 39.12MiB 783.3KiB 16.0KiB 0 0.0195509 False False
http_to_http_json -38.09KiB -0.16 99.68% 23.85MiB 334.8KiB 6.84KiB 0 0.013708 23.81MiB 537.15KiB 10.97KiB 0 0.022027 False False
syslog_log2metric_humio_metrics -48.49KiB -0.37 100.00% 12.86MiB 229.61KiB 4.69KiB 0 0.017429 12.82MiB 482.74KiB 9.83KiB 0 0.0367795 False False
http_to_http_noack -95.27KiB -0.39 100.00% 23.85MiB 249.82KiB 5.11KiB 0 0.0102288 23.75MiB 1.05MiB 21.8KiB 0 0.0439979 False False
fluent_elasticsearch -401.5KiB -0.49 100.00% 79.47MiB 54.2KiB 1.09KiB 0 0.000665931 79.08MiB 4.24MiB 87.15KiB 0 0.0536488 False False
datadog_agent_remap_datadog_logs -428.0KiB -0.68 100.00% 61.73MiB 2.17MiB 45.42KiB 0 0.035093 61.31MiB 4.19MiB 87.36KiB 0 0.0683539 False False
syslog_log2metric_splunk_hec_metrics -132.23KiB -0.73 100.00% 17.69MiB 432.0KiB 8.81KiB 0 0.02384 17.56MiB 788.16KiB 16.04KiB 0 0.0438144 False False
datadog_agent_remap_datadog_logs_acks -549.29KiB -0.89 100.00% 60.51MiB 2.51MiB 52.57KiB 0 0.0414649 59.98MiB 4.37MiB 90.98KiB 0 0.0728573 False False
http_pipelines_blackhole -16.8KiB -0.98 100.00% 1.68MiB 47.54KiB 994.69B 0 0.0276341 1.66MiB 107.76KiB 2.2KiB 0 0.0632618 False False
syslog_loki -203.13KiB -1.36 100.00% 14.6MiB 466.89KiB 9.55KiB 0 0.0312151 14.41MiB 783.48KiB 15.93KiB 0 0.0531023 False False
datadog_agent_remap_blackhole_acks -1.02MiB -1.65 100.00% 62.12MiB 4.23MiB 88.07KiB 0 0.0680643 61.1MiB 4.7MiB 98.13KiB 0 0.0769631 False False

github-actions[bot] avatar Sep 13 '22 16:09 github-actions[bot]

Soak Test Results

Baseline: 1ef12b9ee7c01100f486b151682e8bc60bc3b95f Comparison: 47c012df44cf04a65952b0a2a41575fc85f006a5 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_blackhole_acks 10.72KiB 0.85 99.97% 1.24MiB 113.17KiB 2.3KiB 0 0.0894487 1.25MiB 89.35KiB 1.82KiB 0 0.0700278 False False
splunk_hec_route_s3 104.41KiB 0.54 89.11% 18.82MiB 2.24MiB 46.53KiB 0 0.118767 18.92MiB 2.18MiB 45.54KiB 0 0.115178 False False
datadog_agent_remap_blackhole 233.45KiB 0.38 96.64% 60.72MiB 4.13MiB 85.99KiB 0 0.0679768 60.95MiB 3.27MiB 68.29KiB 0 0.0537192 False False
splunk_hec_to_splunk_hec_logs_noack 15.17KiB 0.06 79.53% 23.82MiB 480.12KiB 9.81KiB 0 0.0196779 23.84MiB 334.95KiB 6.84KiB 0 0.0137197 False False
splunk_hec_to_splunk_hec_logs_acks 11.27KiB 0.05 35.83% 23.75MiB 865.09KiB 17.6KiB 0 0.0355636 23.76MiB 817.15KiB 16.63KiB 0 0.033577 False False
splunk_hec_indexer_ack_blackhole 11.49KiB 0.05 34.83% 23.75MiB 903.87KiB 18.39KiB 0 0.0371608 23.76MiB 864.39KiB 17.59KiB 0 0.0355206 False False
syslog_splunk_hec_logs 8.87KiB 0.05 33.14% 16.12MiB 706.66KiB 14.38KiB 0 0.0428067 16.13MiB 732.58KiB 14.91KiB 0 0.0443532 False False
enterprise_http_to_http -1.65KiB -0.01 18.16% 23.85MiB 246.34KiB 5.03KiB 0 0.0100858 23.85MiB 251.43KiB 5.14KiB 0 0.0102949 False False
syslog_regex_logs2metric_ddmetrics -2.64KiB -0.02 13.26% 12.49MiB 592.27KiB 12.06KiB 0 0.046305 12.49MiB 499.66KiB 10.19KiB 0 0.0390728 False False
file_to_blackhole -49.88KiB -0.05 33.09% 95.33MiB 3.8MiB 78.77KiB 0 0.0398484 95.28MiB 4.14MiB 86.12KiB 0 0.0434401 False False
http_text_to_http_json -53.84KiB -0.13 98.66% 39.39MiB 747.83KiB 15.26KiB 0 0.0185376 39.33MiB 760.38KiB 15.52KiB 0 0.0188739 False False
http_to_http_json -34.76KiB -0.14 99.40% 23.85MiB 335.7KiB 6.85KiB 0 0.0137435 23.81MiB 519.96KiB 10.62KiB 0 0.0213175 False False
syslog_humio_logs -39.21KiB -0.23 100.00% 16.5MiB 134.3KiB 2.74KiB 0 0.00794919 16.46MiB 182.17KiB 3.73KiB 0 0.0108075 False False
fluent_elasticsearch -218.12KiB -0.27 100.00% 79.47MiB 53.51KiB 1.08KiB 0 0.00065738 79.26MiB 1.87MiB 38.51KiB 0 0.02361 False False
syslog_log2metric_humio_metrics -50.09KiB -0.38 100.00% 12.75MiB 182.0KiB 3.72KiB 0 0.0139326 12.71MiB 539.72KiB 10.99KiB 0 0.0414752 False False
http_to_http_noack -95.7KiB -0.39 99.82% 23.8MiB 760.29KiB 15.53KiB 0 0.031186 23.71MiB 1.27MiB 26.44KiB 0 0.0535179 False False
datadog_agent_remap_blackhole_acks -295.46KiB -0.47 99.71% 61.84MiB 4.09MiB 85.28KiB 0 0.0661948 61.55MiB 2.41MiB 50.55KiB 0 0.0392112 False False
http_to_http_acks -99.83KiB -0.56 32.44% 17.46MiB 8.08MiB 168.87KiB 0 0.462613 17.36MiB 8.07MiB 168.5KiB 0 0.464768 True True
datadog_agent_remap_datadog_logs -368.13KiB -0.59 99.99% 61.38MiB 1.17MiB 24.5KiB 0 0.0190488 61.02MiB 4.21MiB 87.7KiB 0 0.0690036 False False
datadog_agent_remap_datadog_logs_acks -393.69KiB -0.62 99.96% 62.16MiB 2.96MiB 61.85KiB 0 0.0475777 61.77MiB 4.38MiB 91.15KiB 0 0.0708722 False False
http_pipelines_blackhole -15.56KiB -0.88 100.00% 1.72MiB 9.22KiB 193.09B 0 0.00524086 1.7MiB 124.68KiB 2.54KiB 0 0.0714851 False False
http_pipelines_no_grok_blackhole -108.56KiB -0.97 100.00% 10.91MiB 51.99KiB 1.06KiB 0 0.004651 10.81MiB 1.07MiB 22.36KiB 0 0.0993266 False False
syslog_loki -149.03KiB -1.01 100.00% 14.35MiB 523.0KiB 10.7KiB 0 0.0355726 14.21MiB 813.52KiB 16.54KiB 0 0.0558993 False False
socket_to_socket_blackhole -337.05KiB -1.45 100.00% 22.7MiB 773.74KiB 15.79KiB 0 0.0332761 22.37MiB 903.89KiB 18.45KiB 0 0.0394456 False False
syslog_log2metric_splunk_hec_metrics -293.91KiB -1.83 100.00% 15.7MiB 1.38MiB 28.8KiB 0 0.0878779 15.41MiB 1.37MiB 28.64KiB 0 0.0891325 False False

github-actions[bot] avatar Sep 13 '22 17:09 github-actions[bot]

Soak Test Results

Baseline: 4186459d72a1f52a7c6476b90733f80f9e553feb Comparison: f25ce8cbedd925957c22169be5a714aba3150ce6 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_blackhole_acks 41.36KiB 3.42 100.00% 1.18MiB 134.05KiB 2.73KiB 0 0.110955 1.22MiB 88.44KiB 1.8KiB 0 0.0707786 False False
syslog_splunk_hec_logs 316.99KiB 1.95 100.00% 15.84MiB 957.75KiB 19.49KiB 0 0.059031 16.15MiB 784.17KiB 15.99KiB 0 0.047406 False False
datadog_agent_remap_blackhole_acks 414.59KiB 0.69 99.98% 58.81MiB 4.27MiB 88.99KiB 0 0.0726372 59.21MiB 3.23MiB 67.64KiB 0 0.0546018 False False
syslog_regex_logs2metric_ddmetrics 71.65KiB 0.55 100.00% 12.62MiB 663.09KiB 13.5KiB 0 0.0512984 12.69MiB 494.71KiB 10.09KiB 0 0.0380613 False False
socket_to_socket_blackhole 100.09KiB 0.42 99.99% 23.28MiB 912.1KiB 18.62KiB 0 0.0382526 23.38MiB 841.98KiB 17.19KiB 0 0.0351642 False False
syslog_log2metric_humio_metrics 49.24KiB 0.38 99.98% 12.72MiB 391.82KiB 7.99KiB 0 0.0300713 12.77MiB 523.39KiB 10.65KiB 0 0.0400183 False False
http_pipelines_blackhole 6.45KiB 0.38 91.82% 1.65MiB 93.4KiB 1.91KiB 0 0.0551792 1.66MiB 156.0KiB 3.18KiB 0 0.0918094 False False
datadog_agent_remap_blackhole 207.44KiB 0.35 98.16% 58.49MiB 3.37MiB 70.2KiB 0 0.0575455 58.69MiB 2.54MiB 53.03KiB 0 0.0432826 False False
syslog_humio_logs 13.26KiB 0.08 98.48% 16.5MiB 188.13KiB 3.84KiB 0 0.0111354 16.51MiB 189.37KiB 3.88KiB 0 0.0111999 False False
syslog_loki 7.52KiB 0.05 32.43% 14.37MiB 495.82KiB 10.15KiB 0 0.033698 14.37MiB 729.93KiB 14.84KiB 0 0.0495844 False False
splunk_hec_indexer_ack_blackhole 10.7KiB 0.04 32.32% 23.75MiB 910.03KiB 18.51KiB 0 0.037414 23.76MiB 873.09KiB 17.77KiB 0 0.0358797 False False
splunk_hec_to_splunk_hec_logs_noack 4.54KiB 0.02 33.74% 23.83MiB 381.03KiB 7.78KiB 0 0.015609 23.84MiB 338.67KiB 6.91KiB 0 0.0138713 False False
syslog_log2metric_splunk_hec_metrics -2.28KiB -0.01 10.87% 17.65MiB 549.27KiB 11.2KiB 0 0.0303825 17.65MiB 605.46KiB 12.34KiB 0 0.033495 False False
splunk_hec_to_splunk_hec_logs_acks -8.2KiB -0.03 25.14% 23.78MiB 880.66KiB 17.92KiB 0 0.036159 23.77MiB 897.15KiB 18.25KiB 0 0.0368483 False False
enterprise_http_to_http -9.3KiB -0.04 65.66% 23.85MiB 251.15KiB 5.13KiB 0 0.010283 23.84MiB 409.27KiB 8.37KiB 0 0.0167634 False False
http_to_http_json -31.59KiB -0.13 98.81% 23.85MiB 342.16KiB 6.98KiB 0 0.0140098 23.82MiB 510.7KiB 10.43KiB 0 0.0209375 False False
file_to_blackhole -143.83KiB -0.15 79.00% 95.35MiB 3.1MiB 64.22KiB 0 0.0325078 95.21MiB 4.58MiB 95.06KiB 0 0.0481232 False False
fluent_elasticsearch -201.11KiB -0.25 100.00% 79.47MiB 53.62KiB 1.08KiB 0 0.000658783 79.28MiB 1.86MiB 38.24KiB 0 0.0234382 False False
http_text_to_http_json -115.25KiB -0.29 99.88% 38.22MiB 1.16MiB 24.19KiB 0 0.030269 38.11MiB 1.25MiB 26.22KiB 0 0.0329014 False False
splunk_hec_route_s3 -56.43KiB -0.3 60.55% 18.55MiB 2.25MiB 46.88KiB 0 0.121209 18.5MiB 2.24MiB 46.83KiB 0 0.121104 False False
http_to_http_noack -76.06KiB -0.31 99.82% 23.83MiB 523.78KiB 10.71KiB 0 0.0214623 23.75MiB 1.05MiB 21.81KiB 0 0.0440282 False False
http_pipelines_no_grok_blackhole -44.68KiB -0.42 95.83% 10.49MiB 337.83KiB 6.9KiB 0 0.0314565 10.44MiB 1023.13KiB 20.82KiB 0 0.0956658 False False
datadog_agent_remap_datadog_logs_acks -989.77KiB -1.63 100.00% 59.27MiB 4.52MiB 94.4KiB 0 0.076183 58.3MiB 5.41MiB 112.64KiB 0 0.0927976 False False
datadog_agent_remap_datadog_logs -1.7MiB -2.76 100.00% 61.77MiB 2.49MiB 52.22KiB 0 0.0403225 60.07MiB 4.94MiB 102.99KiB 0 0.082255 False False
http_to_http_acks -554.09KiB -3.04 98.30% 17.77MiB 7.85MiB 164.02KiB 0 0.441439 17.23MiB 7.86MiB 164.23KiB 0 0.456352 True True

github-actions[bot] avatar Sep 14 '22 19:09 github-actions[bot]

Soak Test Results

Baseline: dec33dc4956fb375fe889d5e41c1db911ec13835 Comparison: 49338ee89de810fa3e94d8abfe246d222f12a6c4 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_datadog_logs 979.26KiB 1.59 100.00% 60.02MiB 3.28MiB 68.76KiB 0 0.0546224 60.98MiB 4.86MiB 101.18KiB 0 0.0796288 False False
datadog_agent_remap_datadog_logs_acks 911.65KiB 1.48 100.00% 60.2MiB 4.29MiB 89.56KiB 0 0.0712171 61.09MiB 4.82MiB 100.31KiB 0 0.0788602 False False
syslog_loki 184.63KiB 1.3 100.00% 13.88MiB 666.63KiB 13.65KiB 0 0.0468938 14.06MiB 815.54KiB 16.58KiB 0 0.0566333 False False
http_text_to_http_json 179.37KiB 0.44 100.00% 39.99MiB 949.15KiB 19.37KiB 0 0.023175 40.16MiB 693.73KiB 14.16KiB 0 0.0168645 False False
http_pipelines_blackhole_acks 3.86KiB 0.31 83.18% 1.23MiB 115.02KiB 2.34KiB 0 0.0914126 1.23MiB 75.14KiB 1.53KiB 0 0.0595316 False False
syslog_regex_logs2metric_ddmetrics 33.23KiB 0.26 95.90% 12.5MiB 562.3KiB 11.46KiB 0 0.0439042 12.54MiB 566.32KiB 11.54KiB 0 0.0441043 False False
syslog_splunk_hec_logs 25.81KiB 0.15 80.74% 16.49MiB 706.65KiB 14.39KiB 0 0.0418329 16.52MiB 667.81KiB 13.61KiB 0 0.0394732 False False
syslog_log2metric_humio_metrics 17.26KiB 0.13 85.72% 12.72MiB 321.92KiB 6.57KiB 0 0.0247088 12.74MiB 479.55KiB 9.77KiB 0 0.036759 False False
splunk_hec_to_splunk_hec_logs_noack 14.67KiB 0.06 79.69% 23.82MiB 453.27KiB 9.25KiB 0 0.0185772 23.84MiB 336.46KiB 6.87KiB 0 0.0137812 False False
splunk_hec_to_splunk_hec_logs_acks 9.25KiB 0.04 29.72% 23.75MiB 859.67KiB 17.48KiB 0 0.0353427 23.76MiB 826.04KiB 16.81KiB 0 0.0339471 False False
syslog_humio_logs 5.55KiB 0.03 72.00% 16.51MiB 179.25KiB 3.66KiB 0 0.0106033 16.51MiB 176.37KiB 3.61KiB 0 0.0104294 False False
enterprise_http_to_http 1.22KiB 0 13.32% 23.84MiB 253.73KiB 5.18KiB 0 0.0103896 23.85MiB 248.93KiB 5.09KiB 0 0.0101926 False False
splunk_hec_indexer_ack_blackhole -11.65KiB -0.05 33.51% 23.75MiB 913.91KiB 18.59KiB 0 0.0375686 23.74MiB 955.53KiB 19.43KiB 0 0.0392985 False False
http_to_http_json -30.16KiB -0.12 98.64% 23.85MiB 336.75KiB 6.87KiB 0 0.0137878 23.82MiB 494.09KiB 10.1KiB 0 0.0202547 False False
file_to_blackhole -142.8KiB -0.15 77.98% 95.33MiB 3.35MiB 69.51KiB 0 0.0351736 95.19MiB 4.5MiB 93.43KiB 0 0.0472785 False False
syslog_log2metric_splunk_hec_metrics -29.34KiB -0.16 81.98% 17.42MiB 675.91KiB 13.78KiB 0 0.037893 17.39MiB 835.04KiB 17.01KiB 0 0.0468917 False False
datadog_agent_remap_blackhole_acks -154.7KiB -0.26 90.24% 58.76MiB 3.66MiB 76.36KiB 0 0.0623461 58.61MiB 2.57MiB 53.72KiB 0 0.0437878 False False
http_to_http_noack -76.81KiB -0.31 99.95% 23.84MiB 404.87KiB 8.28KiB 0 0.0165833 23.76MiB 1010.84KiB 20.59KiB 0 0.0415341 False False
splunk_hec_route_s3 -78.54KiB -0.42 73.45% 18.05MiB 2.39MiB 49.66KiB 0 0.132123 17.98MiB 2.4MiB 50.08KiB 0 0.13324 False False
datadog_agent_remap_blackhole -269.07KiB -0.44 99.32% 59.19MiB 3.77MiB 78.55KiB 0 0.0636463 58.93MiB 2.91MiB 60.74KiB 0 0.049389 False False
socket_to_socket_blackhole -126.54KiB -0.51 100.00% 24.04MiB 145.38KiB 2.97KiB 0 0.00590361 23.92MiB 200.88KiB 4.1KiB 0 0.00819967 False False
fluent_elasticsearch -437.74KiB -0.54 100.00% 79.47MiB 53.37KiB 1.08KiB 0 0.000655701 79.05MiB 4.47MiB 91.85KiB 0 0.056568 False False
http_pipelines_blackhole -11.3KiB -0.66 100.00% 1.67MiB 29.66KiB 621.17B 0 0.0173881 1.65MiB 105.1KiB 2.14KiB 0 0.0620232 False False
http_pipelines_no_grok_blackhole -126.98KiB -1.16 100.00% 10.67MiB 277.85KiB 5.67KiB 0 0.0254144 10.55MiB 1.05MiB 21.96KiB 0 0.0999109 False False
http_to_http_acks -214.98KiB -1.21 64.57% 17.41MiB 8.13MiB 169.95KiB 0 0.46686 17.2MiB 7.56MiB 157.99KiB 0 0.439487 True True

github-actions[bot] avatar Sep 24 '22 00:09 github-actions[bot]