vector icon indicating copy to clipboard operation
vector copied to clipboard

Published 20 hours ago • verified publisher icon vectordotdev

feat(aws_cloudwatch_logs): Add retention policy for cloudwatch logs sink

Open dizlv opened this issue 2 years ago • 4 comments

Solves #12038

dizlv avatar Jan 13 '23 20:01 dizlv

Deploy Preview for vector-project ready!

Name Link
Latest commit 2836c140933fa6ec686f7c56030813654a8e6a22
Latest deploy log https://app.netlify.com/sites/vector-project/deploys/63c5c7cd359aa0000923239a
Deploy Preview https://deploy-preview-15955--vector-project.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify[bot] avatar Jan 13 '23 20:01 netlify[bot]

Deploy Preview for vrl-playground canceled.

Name Link
Latest commit 2836c140933fa6ec686f7c56030813654a8e6a22
Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/63c5c7cd7078740008b795d9

netlify[bot] avatar Jan 13 '23 20:01 netlify[bot]

Regression Test Results

Run ID: b17b2389-fd35-4020-9473-06c5e2a60f35
Baseline: 6d05e13337bf8daacf47f0e289a521408fe079cc
Comparison: 097f499923916f8695051e91d40e04f727bca0b9
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment Δ mean Δ mean % confidence
http_text_to_http_json 1.22MiB/CPU-s 5.05 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json 1.22MiB/CPU-s 5.05 100.00% 24.16MiB/CPU-s 898.54KiB/CPU-s 11.6KiB/CPU-s 0.0 0.036322 25.38MiB/CPU-s 926.21KiB/CPU-s 11.95KiB/CPU-s 0.0 0.035642 False False
datadog_agent_remap_blackhole 1.04MiB/CPU-s 3.48 100.00% 29.96MiB/CPU-s 1.54MiB/CPU-s 20.42KiB/CPU-s 0.0 0.051552 31.01MiB/CPU-s 1.24MiB/CPU-s 16.45KiB/CPU-s 0.0 0.040137 False False
datadog_agent_remap_datadog_logs_acks 904.89KiB/CPU-s 2.62 100.00% 33.69MiB/CPU-s 1.38MiB/CPU-s 18.29KiB/CPU-s 0.0 0.0411 34.57MiB/CPU-s 901.15KiB/CPU-s 11.63KiB/CPU-s 0.0 0.025455 False False
splunk_hec_route_s3 262.15KiB/CPU-s 2.23 100.00% 11.5MiB/CPU-s 588.89KiB/CPU-s 7.6KiB/CPU-s 0.0 0.05 11.76MiB/CPU-s 590.03KiB/CPU-s 7.61KiB/CPU-s 0.0 0.049007 False False
syslog_splunk_hec_logs 164.57KiB/CPU-s 1.82 100.00% 8.85MiB/CPU-s 318.13KiB/CPU-s 4.11KiB/CPU-s 0.0 0.03512 9.01MiB/CPU-s 224.65KiB/CPU-s 2.9KiB/CPU-s 0.0 0.024357 False False
socket_to_socket_blackhole 184.49KiB/CPU-s 1.36 100.00% 13.28MiB/CPU-s 457.14KiB/CPU-s 5.9KiB/CPU-s 0.0 0.03361 13.46MiB/CPU-s 167.67KiB/CPU-s 2.17KiB/CPU-s 0.0 0.012163 False False
datadog_agent_remap_datadog_logs 366.16KiB/CPU-s 1.06 100.00% 33.64MiB/CPU-s 1.67MiB/CPU-s 22.12KiB/CPU-s 0.0 0.049778 34.0MiB/CPU-s 1.86MiB/CPU-s 24.53KiB/CPU-s 0.0 0.054609 False False
syslog_humio_logs 96.83KiB/CPU-s 1.05 100.00% 9.03MiB/CPU-s 261.8KiB/CPU-s 3.38KiB/CPU-s 0.0 0.028318 9.12MiB/CPU-s 151.52KiB/CPU-s 1.96KiB/CPU-s 0.0 0.016219 False False
http_to_http_acks 18.15KiB/CPU-s 0.34 28.08% 5.25MiB/CPU-s 2.71MiB/CPU-s 35.83KiB/CPU-s 0.0 0.516818 5.26MiB/CPU-s 2.69MiB/CPU-s 35.57KiB/CPU-s 0.0 0.511451 True False
enterprise_http_to_http 6.01KiB/CPU-s 0.04 77.55% 13.61MiB/CPU-s 308.48KiB/CPU-s 3.98KiB/CPU-s 0.0 0.022126 13.62MiB/CPU-s 227.84KiB/CPU-s 2.94KiB/CPU-s 0.0 0.016335 False False
http_to_http_noack 6.27KiB/CPU-s 0.04 75.69% 13.61MiB/CPU-s 330.61KiB/CPU-s 4.27KiB/CPU-s 0.0 0.023717 13.62MiB/CPU-s 253.05KiB/CPU-s 3.27KiB/CPU-s 0.0 0.018145 False False
splunk_hec_to_splunk_hec_logs_acks 3.67KiB/CPU-s 0.03 42.59% 13.61MiB/CPU-s 370.17KiB/CPU-s 4.78KiB/CPU-s 0.0 0.02655 13.62MiB/CPU-s 344.64KiB/CPU-s 4.45KiB/CPU-s 0.0 0.024712 False False
splunk_hec_to_splunk_hec_logs_noack 4.69KiB/CPU-s 0.03 77.97% 13.62MiB/CPU-s 228.36KiB/CPU-s 2.95KiB/CPU-s 0.0 0.016372 13.62MiB/CPU-s 188.51KiB/CPU-s 2.43KiB/CPU-s 0.0 0.01351 False False
splunk_hec_indexer_ack_blackhole -604.48B/CPU-s -0.0 10.62% 13.62MiB/CPU-s 241.47KiB/CPU-s 3.12KiB/CPU-s 0.0 0.017314 13.62MiB/CPU-s 243.15KiB/CPU-s 3.14KiB/CPU-s 0.0 0.017435 False False
http_to_http_json 153.11B/CPU-s 0.0 3.10% 13.62MiB/CPU-s 210.26KiB/CPU-s 2.71KiB/CPU-s 0.0 0.015072 13.62MiB/CPU-s 210.89KiB/CPU-s 2.72KiB/CPU-s 0.0 0.015117 False False
fluent_elasticsearch -102.33B/CPU-s -0.0 14.79% 45.41MiB/CPU-s 29.76KiB/CPU-s 389.18B/CPU-s 0.0 0.00064 45.41MiB/CPU-s 29.61KiB/CPU-s 387.23B/CPU-s 0.0 0.000637 False False
file_to_blackhole -11.76KiB/CPU-s -0.02 40.57% 54.5MiB/CPU-s 1.17MiB/CPU-s 15.44KiB/CPU-s 0.0 0.021445 54.49MiB/CPU-s 1.2MiB/CPU-s 15.79KiB/CPU-s 0.0 0.021955 False False
syslog_loki -8.55KiB/CPU-s -0.09 97.99% 8.86MiB/CPU-s 236.54KiB/CPU-s 3.05KiB/CPU-s 0.0 0.026081 8.85MiB/CPU-s 158.96KiB/CPU-s 2.05KiB/CPU-s 0.0 0.017544 False False
syslog_log2metric_splunk_hec_metrics -20.93KiB/CPU-s -0.22 99.91% 9.41MiB/CPU-s 272.2KiB/CPU-s 3.51KiB/CPU-s 0.0 0.028244 9.39MiB/CPU-s 402.44KiB/CPU-s 5.2KiB/CPU-s 0.0 0.04185 False False
otlp_grpc_to_blackhole -3.4KiB/CPU-s -0.32 100.00% 1.05MiB/CPU-s 38.58KiB/CPU-s 510.02B/CPU-s 0.0 0.036021 1.04MiB/CPU-s 47.61KiB/CPU-s 629.21B/CPU-s 0.0 0.044592 False False
otlp_http_to_blackhole -9.34KiB/CPU-s -0.59 100.00% 1.54MiB/CPU-s 106.83KiB/CPU-s 1.38KiB/CPU-s 0.0 0.067725 1.53MiB/CPU-s 113.48KiB/CPU-s 1.46KiB/CPU-s 0.0 0.07237 False False
syslog_log2metric_humio_metrics -81.06KiB/CPU-s -1.31 100.00% 6.07MiB/CPU-s 278.25KiB/CPU-s 3.59KiB/CPU-s 0.0 0.044793 5.99MiB/CPU-s 270.23KiB/CPU-s 3.49KiB/CPU-s 0.0 0.044076 False False
datadog_agent_remap_blackhole_acks -667.32KiB/CPU-s -2.09 100.00% 31.11MiB/CPU-s 1.05MiB/CPU-s 13.89KiB/CPU-s 0.0 0.033767 30.46MiB/CPU-s 1.52MiB/CPU-s 20.04KiB/CPU-s 0.0 0.049766 False False
syslog_regex_logs2metric_ddmetrics -151.77KiB/CPU-s -4.06 100.00% 3.65MiB/CPU-s 415.8KiB/CPU-s 5.37KiB/CPU-s 0.0 0.111288 3.5MiB/CPU-s 439.23KiB/CPU-s 5.67KiB/CPU-s 0.0 0.122536 True False

github-actions[bot] avatar Jan 13 '23 20:01 github-actions[bot]

Regression Test Results

Run ID: f6fc201e-2e04-4ee3-adc4-97faa658dca8
Baseline: 889c167b78c93c2de8b02576f65608e128071110
Comparison: 2836c140933fa6ec686f7c56030813654a8e6a22
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json 998.67KiB/CPU-s 4.09 100.00% 23.83MiB/CPU-s 833.52KiB/CPU-s 10.76KiB/CPU-s 0.0 0.034155 24.81MiB/CPU-s 811.92KiB/CPU-s 10.48KiB/CPU-s 0.0 0.031962 False False
syslog_log2metric_splunk_hec_metrics 279.76KiB/CPU-s 3.08 100.00% 8.86MiB/CPU-s 376.39KiB/CPU-s 4.86KiB/CPU-s 0.0 0.041471 9.14MiB/CPU-s 340.89KiB/CPU-s 4.4KiB/CPU-s 0.0 0.036436 False False
syslog_humio_logs 185.74KiB/CPU-s 1.99 100.00% 9.11MiB/CPU-s 180.91KiB/CPU-s 2.33KiB/CPU-s 0.0 0.019382 9.3MiB/CPU-s 95.38KiB/CPU-s 1.23KiB/CPU-s 0.0 0.010019 False False
syslog_regex_logs2metric_ddmetrics 69.13KiB/CPU-s 1.79 100.00% 3.77MiB/CPU-s 441.22KiB/CPU-s 5.7KiB/CPU-s 0.0 0.11441 3.83MiB/CPU-s 475.49KiB/CPU-s 6.14KiB/CPU-s 0.0 0.121126 True False
socket_to_socket_blackhole 230.81KiB/CPU-s 1.69 100.00% 13.32MiB/CPU-s 190.27KiB/CPU-s 2.46KiB/CPU-s 0.0 0.013953 13.54MiB/CPU-s 181.26KiB/CPU-s 2.34KiB/CPU-s 0.0 0.013071 False False
otlp_http_to_blackhole 23.6KiB/CPU-s 1.53 100.00% 1.51MiB/CPU-s 135.39KiB/CPU-s 1.75KiB/CPU-s 0.0 0.087682 1.53MiB/CPU-s 119.24KiB/CPU-s 1.54KiB/CPU-s 0.0 0.076064 False False
datadog_agent_remap_blackhole_acks 464.94KiB/CPU-s 1.47 100.00% 30.8MiB/CPU-s 1.02MiB/CPU-s 13.5KiB/CPU-s 0.0 0.033155 31.25MiB/CPU-s 932.22KiB/CPU-s 12.03KiB/CPU-s 0.0 0.02913 False False
datadog_agent_remap_datadog_logs_acks 391.44KiB/CPU-s 1.16 100.00% 32.87MiB/CPU-s 1.62MiB/CPU-s 21.35KiB/CPU-s 0.0 0.049172 33.25MiB/CPU-s 1.34MiB/CPU-s 17.68KiB/CPU-s 0.0 0.040234 False False
datadog_agent_remap_datadog_logs 377.53KiB/CPU-s 1.1 100.00% 33.59MiB/CPU-s 1.35MiB/CPU-s 17.85KiB/CPU-s 0.0 0.040221 33.96MiB/CPU-s 1.52MiB/CPU-s 20.05KiB/CPU-s 0.0 0.044685 False False
http_to_http_acks 52.46KiB/CPU-s 0.98 69.38% 5.23MiB/CPU-s 2.71MiB/CPU-s 35.83KiB/CPU-s 0.0 0.518288 5.28MiB/CPU-s 2.77MiB/CPU-s 36.67KiB/CPU-s 0.0 0.525059 True False
otlp_grpc_to_blackhole 9.75KiB/CPU-s 0.92 100.00% 1.03MiB/CPU-s 39.47KiB/CPU-s 521.72B/CPU-s 0.0 0.037279 1.04MiB/CPU-s 51.7KiB/CPU-s 683.07B/CPU-s 0.0 0.048387 False False
syslog_splunk_hec_logs 54.74KiB/CPU-s 0.6 100.00% 8.98MiB/CPU-s 289.03KiB/CPU-s 3.73KiB/CPU-s 0.0 0.031434 9.03MiB/CPU-s 279.34KiB/CPU-s 3.61KiB/CPU-s 0.0 0.030201 False False
enterprise_http_to_http 18.58KiB/CPU-s 0.13 99.94% 13.61MiB/CPU-s 388.84KiB/CPU-s 5.02KiB/CPU-s 0.0 0.027907 13.62MiB/CPU-s 150.29KiB/CPU-s 1.94KiB/CPU-s 0.0 0.010772 False False
syslog_loki 6.44KiB/CPU-s 0.07 82.61% 8.74MiB/CPU-s 258.86KiB/CPU-s 3.34KiB/CPU-s 0.0 0.028916 8.75MiB/CPU-s 260.22KiB/CPU-s 3.36KiB/CPU-s 0.0 0.029048 False False
http_to_http_noack 6.47KiB/CPU-s 0.05 73.45% 13.61MiB/CPU-s 353.22KiB/CPU-s 4.56KiB/CPU-s 0.0 0.025343 13.62MiB/CPU-s 278.96KiB/CPU-s 3.6KiB/CPU-s 0.0 0.020006 False False
splunk_hec_to_splunk_hec_logs_noack 3.63KiB/CPU-s 0.03 66.69% 13.62MiB/CPU-s 218.2KiB/CPU-s 2.82KiB/CPU-s 0.0 0.015642 13.63MiB/CPU-s 192.11KiB/CPU-s 2.48KiB/CPU-s 0.0 0.013768 False False
http_to_http_json -192.86B/CPU-s -0.0 3.89% 13.62MiB/CPU-s 211.29KiB/CPU-s 2.73KiB/CPU-s 0.0 0.015146 13.62MiB/CPU-s 212.1KiB/CPU-s 2.74KiB/CPU-s 0.0 0.015204 False False
fluent_elasticsearch 62.97B/CPU-s 0.0 9.00% 45.41MiB/CPU-s 29.73KiB/CPU-s 388.71B/CPU-s 0.0 0.000639 45.41MiB/CPU-s 30.51KiB/CPU-s 398.77B/CPU-s 0.0 0.000656 False False
file_to_blackhole 992.69B/CPU-s 0.0 3.68% 54.49MiB/CPU-s 1.09MiB/CPU-s 14.36KiB/CPU-s 0.0 0.019949 54.49MiB/CPU-s 1.16MiB/CPU-s 15.31KiB/CPU-s 0.0 0.021281 False False
splunk_hec_to_splunk_hec_logs_acks -3.43KiB/CPU-s -0.02 40.75% 13.62MiB/CPU-s 339.55KiB/CPU-s 4.38KiB/CPU-s 0.0 0.02435 13.61MiB/CPU-s 361.83KiB/CPU-s 4.67KiB/CPU-s 0.0 0.025954 False False
splunk_hec_indexer_ack_blackhole -4.58KiB/CPU-s -0.03 69.08% 13.62MiB/CPU-s 236.45KiB/CPU-s 3.05KiB/CPU-s 0.0 0.016952 13.62MiB/CPU-s 256.49KiB/CPU-s 3.31KiB/CPU-s 0.0 0.018395 False False
splunk_hec_route_s3 -27.5KiB/CPU-s -0.23 98.45% 11.77MiB/CPU-s 562.08KiB/CPU-s 7.25KiB/CPU-s 0.0 0.046616 11.75MiB/CPU-s 677.62KiB/CPU-s 8.74KiB/CPU-s 0.0 0.056326 False False
datadog_agent_remap_blackhole -621.5KiB/CPU-s -2.01 100.00% 30.26MiB/CPU-s 1.87MiB/CPU-s 24.74KiB/CPU-s 0.0 0.061837 29.65MiB/CPU-s 2.25MiB/CPU-s 29.74KiB/CPU-s 0.0 0.07591 False False
syslog_log2metric_humio_metrics -143.1KiB/CPU-s -2.3 100.00% 6.08MiB/CPU-s 179.14KiB/CPU-s 2.31KiB/CPU-s 0.0 0.028795 5.94MiB/CPU-s 210.69KiB/CPU-s 2.72KiB/CPU-s 0.0 0.034662 False False

github-actions[bot] avatar Jan 16 '23 22:01 github-actions[bot]

@dizlv just confirming you're working on this and didn't need any review or feedback yet?

spencergilbert avatar Jan 30 '23 15:01 spencergilbert

@spencergilbert yeah, got burden with regular stuff previous week, most likely will get back on this tomorrow

dizlv avatar Jan 30 '23 21:01 dizlv

@spencergilbert yeah, got burden with regular stuff previous week, most likely will get back on this tomorrow

No rush, just wanted to make sure I wasn't blocking you on something 😅

spencergilbert avatar Jan 30 '23 22:01 spencergilbert

hello @dizlv ! Do plan to continue the development of this feature or maybe you need help?

vtatarin avatar Jul 22 '23 01:07 vtatarin

https://github.com/vectordotdev/vector/pull/18865

AndrewChubatiuk avatar Oct 17 '23 15:10 AndrewChubatiuk

Closed by https://github.com/vectordotdev/vector/pull/18865

jszwedko avatar Dec 01 '23 19:12 jszwedko