If a Kafka cluster rebalances, this consumer could end up pulling data from a different partition, in which case commits caused by acknowledgements will acknowledging the wrong offsets. To handle this, simply flush (drop) all the outstanding finalizers and start over.

This should fix #9587 and #10398

This currently has no integration tests. I am still thinking about how to exercise the new code paths, but wanted to get this out to ensure the basic approach is sane.

Dec 13 '21 22:12 bruceg

Deploy Preview for vector-project canceled.

Name	Link
Latest commit	f93bfbe9ef8b28a0bf2c31da2f3fe5f936ab8468
Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63499e855d5cdf0008d7ba49

Dec 13 '21 22:12 netlify[bot]

I buy the basic approach here, FYI.

Dec 13 '21 22:12 blt

Soak Test Results

Baseline: 962b3fd79445951a914af20e7922e6eb6f0454d8 Comparison: 85e5822eba7c864313b52c02e46122d4dc9bff09 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	baseline outlier percentage	comparison mean	comparison stdev	comparison outlier percentage	t-statistic	p-value	erratic
http_to_http_acks	1.04MiB	19.84	5.23MiB	2.42MiB	0	6.26MiB	2.24MiB	0.828729	-6.01937	2.77336e-09	True
fluent_remap_aws_firehose	701.86KiB	1.54	44.4MiB	951.2KiB	0	45.08MiB	735.06KiB	0.552486	-11.1181	1.67487e-26	False
syslog_loki	534.17KiB	12.31	4.24MiB	759.2KiB	0	4.76MiB	1020.74KiB	0	-7.29427	1.21864e-12	True
datadog_agent_remap_blackhole	128.72KiB	0.66	19.05MiB	382.09KiB	1.5015	19.17MiB	729.21KiB	0	-3.00262	0.00279039	False
syslog_splunk_hec_logs	98.85KiB	1.48	6.53MiB	128.35KiB	0	6.63MiB	214.89KiB	0	-7.1749	2.58097e-12	False
syslog_regex_logs2metric_ddmetrics	58.53KiB	1.22	4.68MiB	286.18KiB	9.66851	4.74MiB	279.58KiB	9.61538	-2.78731	0.00545417	False
syslog_log2metric_splunk_hec_metrics	39.93KiB	0.68	5.76MiB	193.73KiB	0	5.79MiB	61.92KiB	0.547945	-3.75116	0.000199695	False
syslog_log2metric_humio_metrics	36.44KiB	0.68	5.23MiB	146.65KiB	0	5.27MiB	71.75KiB	0.826446	-4.2428	2.61187e-05	False
datadog_agent_remap_datadog_logs	18.86KiB	0.09	21.43MiB	662.71KiB	0	21.45MiB	447.84KiB	0.552486	-0.448724	0.653784	False
http_pipelines_blackhole	1.39KiB	0.24	578.53KiB	575.18KiB	0.877193	579.93KiB	595.13KiB	0.550964	-0.0316232	0.974781	False
http_pipelines_no_grok_blackhole	-12.59KiB	-0.97	1.27MiB	790.07KiB	1.26984	1.26MiB	805.82KiB	0.641026	0.197466	0.843527	False
splunk_transforms_splunk3	-13.2KiB	-0.52	2.48MiB	1.33MiB	1.64835	2.47MiB	1.2MiB	0.292398	0.135212	0.892483	False
syslog_humio_logs	-37.49KiB	-0.54	6.8MiB	170.76KiB	0	6.76MiB	221.41KiB	0	2.55122	0.0109529	False
splunk_hec_route_s3	-121.42KiB	-2.18	5.43MiB	1.91MiB	0.277008	5.31MiB	1.55MiB	0.275482	0.915225	0.360393	False
fluent_elasticsearch	-347.03KiB	-0.68	49.57MiB	925.58KiB	0.826446	49.23MiB	906.63KiB	0	5.09975	4.34909e-07	False
http_to_http_noack	-569.84KiB	-2.7	20.63MiB	1.71MiB	0.879765	20.07MiB	1.73MiB	0	4.17732	3.3459e-05	False

Fine details of each soak run.

(experiment, variant)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_elasticsearch', 'baseline')	363	49.57MiB	925.58KiB	46.74MiB	49.56MiB	50.71MiB	51.02MiB	51.56MiB	51.98MiB	-0.0801949
('fluent_elasticsearch', 'comparison')	362	49.23MiB	906.63KiB	46.83MiB	49.23MiB	50.3MiB	50.67MiB	51.02MiB	51.77MiB	-0.113535
('fluent_remap_aws_firehose', 'comparison')	362	45.08MiB	735.06KiB	43.17MiB	45.07MiB	45.99MiB	46.3MiB	46.89MiB	47.18MiB	0.247433
('fluent_remap_aws_firehose', 'baseline')	363	44.4MiB	951.2KiB	42.24MiB	44.41MiB	45.56MiB	45.85MiB	46.56MiB	46.96MiB	0.0805005
('datadog_agent_remap_datadog_logs', 'comparison')	362	21.45MiB	447.84KiB	20.12MiB	21.42MiB	22.02MiB	22.22MiB	22.39MiB	22.87MiB	0.155508
('datadog_agent_remap_datadog_logs', 'baseline')	362	21.43MiB	662.71KiB	19.96MiB	21.41MiB	22.3MiB	22.47MiB	22.72MiB	23.04MiB	0.0378994
('http_to_http_noack', 'baseline')	341	20.63MiB	1.71MiB	15.4MiB	20.73MiB	22.7MiB	23.18MiB	23.98MiB	25.59MiB	-0.228046
('http_to_http_noack', 'comparison')	324	20.07MiB	1.73MiB	15.94MiB	19.96MiB	22.35MiB	22.87MiB	23.84MiB	23.98MiB	0.0736374
('datadog_agent_remap_blackhole', 'comparison')	380	19.17MiB	729.21KiB	17.42MiB	19.22MiB	20.08MiB	20.22MiB	20.45MiB	20.77MiB	-0.149244
('datadog_agent_remap_blackhole', 'baseline')	333	19.05MiB	382.09KiB	17.85MiB	19.05MiB	19.53MiB	19.68MiB	19.95MiB	20.43MiB	0.13422
('syslog_humio_logs', 'baseline')	362	6.8MiB	170.76KiB	6.44MiB	6.8MiB	7.01MiB	7.04MiB	7.09MiB	7.13MiB	0.0434038
('syslog_humio_logs', 'comparison')	362	6.76MiB	221.41KiB	6.34MiB	6.74MiB	7.04MiB	7.09MiB	7.14MiB	7.16MiB	0.0703287
('syslog_splunk_hec_logs', 'comparison')	320	6.63MiB	214.89KiB	6.27MiB	6.54MiB	6.91MiB	6.94MiB	6.99MiB	7.05MiB	0.273078
('syslog_splunk_hec_logs', 'baseline')	362	6.53MiB	128.35KiB	6.26MiB	6.54MiB	6.69MiB	6.71MiB	6.75MiB	6.8MiB	-0.0901325
('http_to_http_acks', 'comparison')	362	6.26MiB	2.24MiB	0B	6.66MiB	9.41MiB	9.83MiB	10.28MiB	11.89MiB	-0.252214
('syslog_log2metric_splunk_hec_metrics', 'comparison')	365	5.79MiB	61.92KiB	5.62MiB	5.79MiB	5.88MiB	5.89MiB	5.92MiB	5.95MiB	-0.0415491
('syslog_log2metric_splunk_hec_metrics', 'baseline')	365	5.76MiB	193.73KiB	5.46MiB	5.78MiB	5.99MiB	6.01MiB	6.06MiB	6.11MiB	0.0270822
('splunk_hec_route_s3', 'baseline')	361	5.43MiB	1.91MiB	1.07MiB	5.39MiB	7.86MiB	8.49MiB	10.24MiB	11.17MiB	0.154656
('splunk_hec_route_s3', 'comparison')	363	5.31MiB	1.55MiB	611.99KiB	5.45MiB	7.29MiB	7.83MiB	8.69MiB	9.24MiB	-0.136062
('syslog_log2metric_humio_metrics', 'comparison')	363	5.27MiB	71.75KiB	5.06MiB	5.27MiB	5.36MiB	5.38MiB	5.43MiB	5.48MiB	-0.089022
('syslog_log2metric_humio_metrics', 'baseline')	361	5.23MiB	146.65KiB	4.94MiB	5.23MiB	5.42MiB	5.47MiB	5.53MiB	5.58MiB	0.167054
('http_to_http_acks', 'baseline')	371	5.23MiB	2.42MiB	1.4MiB	4.92MiB	8.85MiB	9.37MiB	10.26MiB	10.72MiB	0.224724
('syslog_loki', 'comparison')	276	4.76MiB	1020.74KiB	3.17MiB	4.84MiB	6.13MiB	6.25MiB	6.42MiB	6.6MiB	0.0911268
('syslog_regex_logs2metric_ddmetrics', 'comparison')	364	4.74MiB	279.58KiB	4.43MiB	4.63MiB	5.21MiB	5.38MiB	5.48MiB	5.53MiB	1.41759
('syslog_regex_logs2metric_ddmetrics', 'baseline')	362	4.68MiB	286.18KiB	4.29MiB	4.57MiB	5.22MiB	5.31MiB	5.39MiB	5.51MiB	1.32682
('syslog_loki', 'baseline')	363	4.24MiB	759.2KiB	3.04MiB	4.21MiB	5.19MiB	5.57MiB	5.87MiB	6.03MiB	0.274307
('splunk_transforms_splunk3', 'baseline')	364	2.48MiB	1.33MiB	0B	2.37MiB	4.32MiB	5.01MiB	5.99MiB	7.24MiB	0.55162
('splunk_transforms_splunk3', 'comparison')	342	2.47MiB	1.2MiB	243.88KiB	2.44MiB	3.99MiB	4.64MiB	5.45MiB	6.2MiB	0.412456
('http_pipelines_no_grok_blackhole', 'baseline')	315	1.27MiB	790.07KiB	0B	1.2MiB	2.29MiB	2.65MiB	3.49MiB	3.94MiB	0.589006
('http_pipelines_no_grok_blackhole', 'comparison')	312	1.26MiB	805.82KiB	0B	1.08MiB	2.3MiB	2.67MiB	3.49MiB	4.15MiB	0.798033
('http_pipelines_blackhole', 'comparison')	363	579.93KiB	595.13KiB	0B	432.52KiB	1.46MiB	1.75MiB	2.16MiB	2.45MiB	1.06317
('http_pipelines_blackhole', 'baseline')	342	578.53KiB	575.18KiB	0B	371.3KiB	1.31MiB	1.67MiB	2.09MiB	3.35MiB	1.3792

Dec 13 '21 23:12 github-actions[bot]

Soak Test Results

Baseline: 962b3fd79445951a914af20e7922e6eb6f0454d8 Comparison: 6bcfaac8fa93e92810277c5edfb4a06c47b4cb54 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	baseline outlier percentage	comparison mean	comparison stdev	comparison outlier percentage	t-statistic	p-value	erratic
http_to_http_noack	615.02KiB	2.96	20.31MiB	1.59MiB	1.10193	20.91MiB	1.51MiB	1.0989	-5.22941	2.22877e-07	False
syslog_humio_logs	276.56KiB	4.1	6.59MiB	62.9KiB	0.293255	6.86MiB	99.04KiB	0.276243	-44.456	1.94574e-194	False
syslog_log2metric_splunk_hec_metrics	60.64KiB	1	5.93MiB	139.91KiB	0	5.99MiB	74.25KiB	0.547945	-7.29111	1.07995e-12	False
splunk_hec_route_s3	57.69KiB	1.05	5.36MiB	2.09MiB	1.37741	5.41MiB	2.02MiB	0.310559	-0.359091	0.719639	False
syslog_log2metric_humio_metrics	46.33KiB	0.88	5.16MiB	151.06KiB	0	5.21MiB	102.04KiB	0	-4.78941	2.08255e-06	False
datadog_agent_remap_blackhole	41.71KiB	0.22	18.86MiB	755.66KiB	0	18.9MiB	688.71KiB	0.593472	-0.774186	0.439076	False
splunk_transforms_splunk3	13.03KiB	0.53	2.41MiB	1.14MiB	0.276243	2.42MiB	1.32MiB	1.38122	-0.138379	0.88998	False
http_pipelines_blackhole	-7.2KiB	-1.27	566.52KiB	611.97KiB	1.0989	559.33KiB	579.11KiB	0.273224	0.163214	0.870396	False
syslog_regex_logs2metric_ddmetrics	-7.37KiB	-0.15	4.87MiB	269.86KiB	13.1507	4.87MiB	257.09KiB	13.7427	0.371936	0.710052	False
http_pipelines_no_grok_blackhole	-33.38KiB	-2.6	1.26MiB	832.47KiB	0.554017	1.22MiB	736.73KiB	1.26582	0.553409	0.580167	False
syslog_splunk_hec_logs	-189.8KiB	-2.78	6.67MiB	277.8KiB	0	6.48MiB	61.06KiB	0.552486	12.0405	3.48535e-28	False
syslog_loki	-252.19KiB	-5.98	4.12MiB	783.52KiB	0	3.87MiB	607.45KiB	2.74725	4.43183	1.14748e-05	True
datadog_agent_remap_datadog_logs	-294.08KiB	-1.34	21.36MiB	583.22KiB	0	21.07MiB	401.08KiB	0.552486	7.94892	8.30937e-15	False
fluent_remap_aws_firehose	-629.12KiB	-1.34	45.88MiB	1.13MiB	0	45.27MiB	832.36KiB	1.24611	7.70392	6.46221e-14	False
http_to_http_acks	-874.21KiB	-14.51	5.88MiB	2.39MiB	0	5.03MiB	2.3MiB	0	4.90128	1.17727e-06	True
fluent_elasticsearch	-940.45KiB	-1.83	50.1MiB	1.59MiB	0	49.18MiB	1012.78KiB	0.58651	9.28204	2.82562e-19	False

Fine details of each soak run.

(experiment, variant)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_elasticsearch', 'baseline')	364	50.1MiB	1.59MiB	45.77MiB	50.09MiB	52.13MiB	52.57MiB	53.07MiB	53.57MiB	-0.120683
('fluent_elasticsearch', 'comparison')	341	49.18MiB	1012.78KiB	45.84MiB	49.19MiB	50.53MiB	50.8MiB	51.5MiB	52.22MiB	0.0836005
('fluent_remap_aws_firehose', 'baseline')	297	45.88MiB	1.13MiB	42.63MiB	46.0MiB	47.27MiB	47.61MiB	48.04MiB	48.33MiB	-0.307983
('fluent_remap_aws_firehose', 'comparison')	321	45.27MiB	832.36KiB	42.23MiB	45.36MiB	46.24MiB	46.42MiB	46.67MiB	47.26MiB	-0.524361
('datadog_agent_remap_datadog_logs', 'baseline')	368	21.36MiB	583.22KiB	20.1MiB	21.31MiB	22.1MiB	22.22MiB	22.47MiB	22.87MiB	-0.00537364
('datadog_agent_remap_datadog_logs', 'comparison')	362	21.07MiB	401.08KiB	19.84MiB	21.08MiB	21.55MiB	21.69MiB	22.04MiB	22.44MiB	0.0767162
('http_to_http_noack', 'comparison')	364	20.91MiB	1.51MiB	16.47MiB	20.9MiB	22.94MiB	23.29MiB	23.97MiB	25.38MiB	-0.107375
('http_to_http_noack', 'baseline')	363	20.31MiB	1.59MiB	15.73MiB	20.31MiB	22.16MiB	22.99MiB	24.25MiB	24.81MiB	-0.0390686
('datadog_agent_remap_blackhole', 'comparison')	337	18.9MiB	688.71KiB	17.27MiB	18.8MiB	19.88MiB	20.14MiB	20.38MiB	20.73MiB	0.309531
('datadog_agent_remap_blackhole', 'baseline')	382	18.86MiB	755.66KiB	17.27MiB	18.83MiB	19.85MiB	20.04MiB	20.25MiB	20.64MiB	0.0847685
('syslog_humio_logs', 'comparison')	362	6.86MiB	99.04KiB	6.66MiB	6.85MiB	7.0MiB	7.04MiB	7.07MiB	7.15MiB	0.388407
('syslog_splunk_hec_logs', 'baseline')	324	6.67MiB	277.8KiB	6.21MiB	6.8MiB	6.96MiB	6.98MiB	7.03MiB	7.06MiB	-0.261135
('syslog_humio_logs', 'baseline')	341	6.59MiB	62.9KiB	6.39MiB	6.59MiB	6.67MiB	6.69MiB	6.72MiB	6.75MiB	-0.0730066
('syslog_splunk_hec_logs', 'comparison')	362	6.48MiB	61.06KiB	6.24MiB	6.48MiB	6.55MiB	6.57MiB	6.6MiB	6.62MiB	-0.458893
('syslog_log2metric_splunk_hec_metrics', 'comparison')	365	5.99MiB	74.25KiB	5.78MiB	6.0MiB	6.08MiB	6.11MiB	6.14MiB	6.21MiB	-0.209164
('syslog_log2metric_splunk_hec_metrics', 'baseline')	362	5.93MiB	139.91KiB	5.69MiB	5.91MiB	6.1MiB	6.13MiB	6.15MiB	6.17MiB	0.0371108
('http_to_http_acks', 'baseline')	361	5.88MiB	2.39MiB	1.25MiB	6.58MiB	9.15MiB	9.71MiB	9.97MiB	11.02MiB	-0.140754
('splunk_hec_route_s3', 'comparison')	322	5.41MiB	2.02MiB	1.16MiB	5.26MiB	7.95MiB	8.84MiB	10.36MiB	11.71MiB	0.28885
('splunk_hec_route_s3', 'baseline')	363	5.36MiB	2.09MiB	884.71KiB	5.01MiB	8.3MiB	8.99MiB	11.18MiB	11.86MiB	0.594547
('syslog_log2metric_humio_metrics', 'comparison')	341	5.21MiB	102.04KiB	4.97MiB	5.22MiB	5.33MiB	5.37MiB	5.41MiB	5.45MiB	-0.0179227
('syslog_log2metric_humio_metrics', 'baseline')	362	5.16MiB	151.06KiB	4.88MiB	5.16MiB	5.36MiB	5.38MiB	5.43MiB	5.45MiB	0.035331
('http_to_http_acks', 'comparison')	362	5.03MiB	2.3MiB	0B	4.81MiB	7.68MiB	8.33MiB	10.38MiB	11.08MiB	0.135396
('syslog_regex_logs2metric_ddmetrics', 'baseline')	365	4.87MiB	269.86KiB	4.51MiB	4.8MiB	5.35MiB	5.47MiB	5.61MiB	5.66MiB	1.36069
('syslog_regex_logs2metric_ddmetrics', 'comparison')	342	4.87MiB	257.09KiB	4.51MiB	4.78MiB	5.36MiB	5.41MiB	5.55MiB	5.64MiB	1.39751
('syslog_loki', 'baseline')	276	4.12MiB	783.52KiB	3.17MiB	3.77MiB	5.23MiB	5.68MiB	6.02MiB	6.23MiB	0.857372
('syslog_loki', 'comparison')	364	3.87MiB	607.45KiB	3.09MiB	3.67MiB	4.71MiB	5.01MiB	5.89MiB	6.46MiB	1.41086
('splunk_transforms_splunk3', 'comparison')	362	2.42MiB	1.32MiB	213.45KiB	2.19MiB	4.26MiB	4.86MiB	6.17MiB	7.87MiB	0.845094
('splunk_transforms_splunk3', 'baseline')	362	2.41MiB	1.14MiB	121.94KiB	2.26MiB	3.99MiB	4.5MiB	5.2MiB	5.69MiB	0.44339
('http_pipelines_no_grok_blackhole', 'baseline')	361	1.26MiB	832.47KiB	0B	1.14MiB	2.39MiB	2.84MiB	3.33MiB	4.09MiB	0.710971
('http_pipelines_no_grok_blackhole', 'comparison')	316	1.22MiB	736.73KiB	0B	1.18MiB	2.16MiB	2.43MiB	3.29MiB	3.88MiB	0.601291
('http_pipelines_blackhole', 'baseline')	364	566.52KiB	611.97KiB	0B	308.13KiB	1.44MiB	1.67MiB	2.29MiB	2.87MiB	1.19821
('http_pipelines_blackhole', 'comparison')	366	559.33KiB	579.11KiB	0B	370.74KiB	1.43MiB	1.67MiB	1.97MiB	3.37MiB	1.21544

Dec 14 '21 03:12 github-actions[bot]

@tobz The testing changes I mentioned are in eb9744f. Right now both connect, and both pull down the same 100 events. The resulting offset varies somewhere between 98 and 200 depending on the timing delay I put into the pipeline, despite the messages logged to stderr showing the numbers 0-99. If I use collect_ready instead of collect_n, no events are pulled out of the pipeline, which makes sense because it's only driving the pipeline with the collect that causes the delays to actually wait. I've tried so many different things I can no longer see a path to actually getting this tested.

Dec 16 '21 21:12 bruceg

Soak Test Results

Baseline: c71fa500635b2904d6baf5f92e4223f2f46ed99c Comparison: eb9744f1adca63cd9cff241d30d91a091d2ca0b1 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	baseline outlier percentage	comparison mean	comparison stdev	comparison outlier percentage	t-statistic	p-value	erratic
fluent_elasticsearch	1.65MiB	2.87	57.5MiB	621.73KiB	0	59.14MiB	848.9KiB	0.276243	-30.5107	2.53066e-128	False
datadog_agent_remap_blackhole	616.04KiB	2.17	27.68MiB	405.43KiB	0.526316	28.28MiB	472.03KiB	0.260417	-19.3574	5.66804e-68	False
http_to_http_acks	449.06KiB	8.15	5.38MiB	2.6MiB	0	5.82MiB	2.37MiB	0	-2.37652	0.0177374	True
syslog_log2metric_splunk_hec_metrics	154.71KiB	2.13	7.1MiB	133.4KiB	0	7.25MiB	167.25KiB	0	-13.7741	2.66925e-38	False
syslog_splunk_hec_logs	61.33KiB	0.85	7.08MiB	106.32KiB	0	7.14MiB	80.59KiB	0	-8.7779	1.34447e-17	False
http_pipelines_blackhole_acks	50.64KiB	6.4	790.69KiB	676.36KiB	0.549451	841.32KiB	646.43KiB	0.738007	-0.957207	0.338852	False
http_datadog_filter_blackhole	48.41KiB	3.73	1.27MiB	837.07KiB	1.11111	1.31MiB	841.48KiB	1.10497	-0.774994	0.438598	False
splunk_hec_to_splunk_hec_logs_acks	-577.49B	-2.57	21.94KiB	123.98KiB	6.06061	21.37KiB	102.89KiB	6.59341	0.0667263	0.946819	False
http_pipelines_blackhole	-9.6KiB	-1.16	828.23KiB	622.17KiB	1.37363	818.63KiB	688.7KiB	1.38504	0.196826	0.84402	False
syslog_loki	-22.68KiB	-0.32	6.84MiB	110.23KiB	0	6.82MiB	226.83KiB	0	1.71503	0.0869282	True
http_pipelines_no_grok_blackhole	-37.13KiB	-1.91	1.9MiB	1002.58KiB	0.315457	1.87MiB	956.64KiB	1.09589	0.492713	0.62238	False
syslog_humio_logs	-40.49KiB	-0.54	7.27MiB	173.76KiB	0	7.23MiB	195.87KiB	0	2.80784	0.00513981	False
splunk_hec_to_splunk_hec_logs_noack	-73.76KiB	-0.42	17.22MiB	1.18MiB	1.10497	17.15MiB	1.38MiB	0.550964	0.754897	0.450563	False
datadog_agent_remap_datadog_logs	-107.0KiB	-0.3	34.73MiB	1.14MiB	0	34.62MiB	1001.78KiB	0	1.32212	0.186558	False
splunk_transforms_splunk3	-110.76KiB	-2.59	4.18MiB	1.33MiB	1.88679	4.07MiB	1.5MiB	1.65746	0.998201	0.318538	False
datadog_agent_remap_datadog_logs_acks	-141.82KiB	-0.36	37.97MiB	482.44KiB	0.549451	37.83MiB	761.41KiB	0	3.01049	0.00271432	False
syslog_log2metric_humio_metrics	-151.32KiB	-2.07	7.15MiB	249.32KiB	0	7.0MiB	64.2KiB	0	11.1824	1.65023e-25	False
syslog_regex_logs2metric_ddmetrics	-195.61KiB	-3.07	6.21MiB	165.04KiB	14.3695	6.02MiB	272.05KiB	0	11.6007	3.22921e-28	False
splunk_hec_route_s3	-516.22KiB	-6.21	8.12MiB	2.22MiB	0.554017	7.61MiB	2.16MiB	1.10193	3.09321	0.00205641	False
datadog_agent_remap_blackhole_acks	-624.52KiB	-2.14	28.44MiB	970.46KiB	0	27.83MiB	526.25KiB	0	11.05	6.37706e-26	False
splunk_hec_indexer_ack_blackhole	-700.37KiB	-3.06	22.38MiB	1.28MiB	1.38889	21.69MiB	1.21MiB	1.93906	7.36935	4.73779e-13	False
http_to_http_noack	-1.39MiB	-6.35	21.95MiB	1.55MiB	1.10497	20.55MiB	1.73MiB	0.554017	11.4054	8.62165e-28	False
fluent_remap_aws_firehose	-3.96MiB	-6.57	60.39MiB	1.41MiB	0	56.43MiB	1013.21KiB	0	43.7602	1.11822e-195	False

Fine details of each soak run.

(experiment, variant)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_remap_aws_firehose', 'baseline')	363	60.39MiB	1.41MiB	57.63MiB	60.33MiB	62.07MiB	62.24MiB	62.58MiB	62.74MiB	-0.0845809
('fluent_elasticsearch', 'comparison')	362	59.14MiB	848.9KiB	56.68MiB	59.14MiB	60.25MiB	60.42MiB	60.98MiB	61.4MiB	-0.105981
('fluent_elasticsearch', 'baseline')	362	57.5MiB	621.73KiB	55.84MiB	57.49MiB	58.32MiB	58.5MiB	58.83MiB	59.02MiB	0.0555834
('fluent_remap_aws_firehose', 'comparison')	362	56.43MiB	1013.21KiB	54.0MiB	56.3MiB	57.91MiB	58.23MiB	58.6MiB	58.8MiB	0.253214
('datadog_agent_remap_datadog_logs_acks', 'baseline')	364	37.97MiB	482.44KiB	36.59MiB	37.94MiB	38.61MiB	38.76MiB	39.05MiB	39.38MiB	0.21928
('datadog_agent_remap_datadog_logs_acks', 'comparison')	367	37.83MiB	761.41KiB	35.95MiB	37.8MiB	38.8MiB	38.99MiB	39.25MiB	39.75MiB	-0.0418942
('datadog_agent_remap_datadog_logs', 'baseline')	362	34.73MiB	1.14MiB	31.58MiB	34.93MiB	36.07MiB	36.28MiB	36.88MiB	37.11MiB	-0.257375
('datadog_agent_remap_datadog_logs', 'comparison')	364	34.62MiB	1001.78KiB	32.4MiB	34.43MiB	36.0MiB	36.32MiB	36.83MiB	37.1MiB	0.301184
('datadog_agent_remap_blackhole_acks', 'baseline')	382	28.44MiB	970.46KiB	26.59MiB	28.43MiB	29.7MiB	29.88MiB	30.16MiB	30.42MiB	0.109334
('datadog_agent_remap_blackhole', 'comparison')	384	28.28MiB	472.03KiB	26.98MiB	28.28MiB	28.87MiB	29.0MiB	29.36MiB	29.55MiB	-0.0148461
('datadog_agent_remap_blackhole_acks', 'comparison')	380	27.83MiB	526.25KiB	26.5MiB	27.78MiB	28.5MiB	28.66MiB	28.93MiB	29.15MiB	0.108312
('datadog_agent_remap_blackhole', 'baseline')	380	27.68MiB	405.43KiB	26.54MiB	27.68MiB	28.2MiB	28.32MiB	28.6MiB	28.91MiB	0.0321801
('splunk_hec_indexer_ack_blackhole', 'baseline')	360	22.38MiB	1.28MiB	18.74MiB	22.22MiB	24.17MiB	24.76MiB	25.4MiB	26.43MiB	0.334466
('http_to_http_noack', 'baseline')	362	21.95MiB	1.55MiB	17.72MiB	21.89MiB	23.83MiB	24.51MiB	26.39MiB	27.31MiB	0.331132
('splunk_hec_indexer_ack_blackhole', 'comparison')	361	21.69MiB	1.21MiB	17.52MiB	21.81MiB	23.12MiB	23.48MiB	23.98MiB	24.58MiB	-0.558812
('http_to_http_noack', 'comparison')	361	20.55MiB	1.73MiB	15.46MiB	20.58MiB	22.84MiB	23.14MiB	24.1MiB	24.87MiB	-0.199938
('splunk_hec_to_splunk_hec_logs_noack', 'baseline')	362	17.22MiB	1.18MiB	13.98MiB	17.22MiB	18.77MiB	19.22MiB	19.92MiB	20.59MiB	0.0412879
('splunk_hec_to_splunk_hec_logs_noack', 'comparison')	363	17.15MiB	1.38MiB	14.03MiB	17.16MiB	19.0MiB	19.46MiB	20.58MiB	21.31MiB	0.285188
('splunk_hec_route_s3', 'baseline')	361	8.12MiB	2.22MiB	1.55MiB	8.25MiB	10.87MiB	11.5MiB	12.88MiB	15.15MiB	-0.047006
('splunk_hec_route_s3', 'comparison')	363	7.61MiB	2.16MiB	2.17MiB	7.42MiB	10.39MiB	11.4MiB	13.16MiB	14.45MiB	0.27196
('syslog_humio_logs', 'baseline')	343	7.27MiB	173.76KiB	6.93MiB	7.2MiB	7.48MiB	7.52MiB	7.56MiB	7.6MiB	0.141825
('syslog_log2metric_splunk_hec_metrics', 'comparison')	362	7.25MiB	167.25KiB	6.98MiB	7.25MiB	7.45MiB	7.47MiB	7.49MiB	7.52MiB	0.00858883
('syslog_humio_logs', 'comparison')	320	7.23MiB	195.87KiB	6.91MiB	7.16MiB	7.48MiB	7.52MiB	7.57MiB	7.62MiB	0.216344
('syslog_log2metric_humio_metrics', 'baseline')	362	7.15MiB	249.32KiB	6.79MiB	7.14MiB	7.43MiB	7.45MiB	7.5MiB	7.52MiB	0.0113592
('syslog_splunk_hec_logs', 'comparison')	364	7.14MiB	80.59KiB	6.92MiB	7.14MiB	7.25MiB	7.27MiB	7.29MiB	7.33MiB	-0.135859
('syslog_log2metric_splunk_hec_metrics', 'baseline')	364	7.1MiB	133.4KiB	6.87MiB	7.1MiB	7.26MiB	7.28MiB	7.31MiB	7.31MiB	-0.0651961
('syslog_splunk_hec_logs', 'baseline')	365	7.08MiB	106.32KiB	6.89MiB	7.08MiB	7.21MiB	7.24MiB	7.28MiB	7.32MiB	0.0899936
('syslog_log2metric_humio_metrics', 'comparison')	362	7.0MiB	64.2KiB	6.83MiB	7.0MiB	7.07MiB	7.1MiB	7.13MiB	7.14MiB	-0.161116
('syslog_loki', 'baseline')	363	6.84MiB	110.23KiB	6.54MiB	6.85MiB	6.97MiB	6.99MiB	7.03MiB	7.07MiB	-0.280811
('syslog_loki', 'comparison')	364	6.82MiB	226.83KiB	6.29MiB	6.9MiB	7.06MiB	7.07MiB	7.12MiB	7.14MiB	-0.592
('syslog_regex_logs2metric_ddmetrics', 'baseline')	341	6.21MiB	165.04KiB	5.72MiB	6.26MiB	6.36MiB	6.38MiB	6.43MiB	6.48MiB	-1.42957
('syslog_regex_logs2metric_ddmetrics', 'comparison')	362	6.02MiB	272.05KiB	5.67MiB	5.87MiB	6.38MiB	6.41MiB	6.46MiB	6.47MiB	0.312931
('http_to_http_acks', 'comparison')	363	5.82MiB	2.37MiB	1.22MiB	6.02MiB	9.31MiB	9.68MiB	10.67MiB	11.86MiB	0.155007
('http_to_http_acks', 'baseline')	364	5.38MiB	2.6MiB	1.07MiB	5.53MiB	9.18MiB	9.56MiB	10.36MiB	12.03MiB	0.0836072
('splunk_transforms_splunk3', 'baseline')	318	4.18MiB	1.33MiB	1.19MiB	4.04MiB	5.87MiB	6.42MiB	7.7MiB	8.67MiB	0.47689
('splunk_transforms_splunk3', 'comparison')	362	4.07MiB	1.5MiB	640.4KiB	3.9MiB	6.05MiB	6.64MiB	8.51MiB	9.38MiB	0.651511
('http_pipelines_no_grok_blackhole', 'baseline')	317	1.9MiB	1002.58KiB	186.49KiB	1.82MiB	3.28MiB	3.74MiB	4.36MiB	5.02MiB	0.534761
('http_pipelines_no_grok_blackhole', 'comparison')	365	1.87MiB	956.64KiB	31.98KiB	1.79MiB	3.08MiB	3.55MiB	4.29MiB	4.65MiB	0.367158
('http_datadog_filter_blackhole', 'comparison')	362	1.31MiB	841.48KiB	0B	1.24MiB	2.48MiB	2.87MiB	3.56MiB	4.12MiB	0.655597
('http_datadog_filter_blackhole', 'baseline')	360	1.27MiB	837.07KiB	0B	1.19MiB	2.39MiB	2.75MiB	3.63MiB	4.06MiB	0.654639
('http_pipelines_blackhole_acks', 'comparison')	271	841.32KiB	646.43KiB	0B	741.97KiB	1.62MiB	1.9MiB	2.54MiB	3.97MiB	0.950849
('http_pipelines_blackhole', 'baseline')	364	828.23KiB	622.17KiB	0B	694.55KiB	1.58MiB	2.02MiB	2.61MiB	2.89MiB	0.809656
('http_pipelines_blackhole', 'comparison')	361	818.63KiB	688.7KiB	0B	711.32KiB	1.71MiB	2.09MiB	2.63MiB	3.42MiB	0.999494
('http_pipelines_blackhole_acks', 'baseline')	364	790.69KiB	676.36KiB	0B	615.74KiB	1.62MiB	2.03MiB	2.61MiB	3.32MiB	0.834152
('splunk_hec_to_splunk_hec_logs_acks', 'baseline')	363	21.94KiB	123.98KiB	0B	0B	0B	30.47KiB	825.06KiB	1.1MiB	6.97952
('splunk_hec_to_splunk_hec_logs_acks', 'comparison')	364	21.37KiB	102.89KiB	0B	0B	0B	112.82KiB	640.71KiB	793.37KiB	5.65279

Dec 16 '21 22:12 github-actions[bot]

I tracked down the rdkafka bug: https://github.com/fede1024/rust-rdkafka/issues/416

Dec 29 '21 22:12 bruceg

Soak Test Results

Baseline: 9fecdc8b5c45c613de2d01d4d2aee22be3a2e570 Comparison: 5352c59dff8f3ad936ddde014336db6088ca22dd Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	baseline outlier percentage	comparison mean	comparison stdev	comparison outlier percentage	t-statistic	p-value	erratic
datadog_agent_remap_datadog_logs	1.91MiB	5.3	36.08MiB	710.2KiB	0	37.99MiB	1.41MiB	0	-23.1669	3.09128e-82	False
datadog_agent_remap_blackhole	1.28MiB	4.37	29.21MiB	914.72KiB	0	30.48MiB	413.22KiB	1.31579	-25.3582	2.34026e-93	False
datadog_agent_remap_blackhole_acks	1.1MiB	3.71	29.7MiB	993.83KiB	0	30.8MiB	672.54KiB	0	-18.3456	3.7982e-61	False
http_to_http_acks	737.91KiB	9.65	7.47MiB	6.23MiB	0	8.19MiB	5.79MiB	0	-1.61399	0.106967	True
http_to_http_noack	387.19KiB	1.75	21.55MiB	1.35MiB	1.10193	21.93MiB	1.66MiB	0.547945	-3.37126	0.00078952	False
fluent_elasticsearch	268.07KiB	0.4	64.9MiB	609.2KiB	0.552486	65.16MiB	817.36KiB	0.275482	-5.00771	7.05324e-07	False
syslog_splunk_hec_logs	150.59KiB	1.66	8.87MiB	200.86KiB	0	9.01MiB	179.3KiB	0	-10.6482	1.12984e-24	False
splunk_transforms_splunk3	135.37KiB	2.96	4.47MiB	1.65MiB	0.828729	4.6MiB	1.52MiB	0.277008	-1.12177	0.262335	False
syslog_loki	48.34KiB	0.66	7.2MiB	182.38KiB	0	7.25MiB	250.21KiB	15.1515	-2.94306	0.00336363	True
http_pipelines_blackhole_acks	45.73KiB	5.4	846.08KiB	742.51KiB	1.89873	891.81KiB	820.47KiB	3.26975	-0.764338	0.444931	False
http_pipelines_blackhole	38.64KiB	4.57	845.96KiB	706.42KiB	3.45745	884.6KiB	768.38KiB	1.50376	-0.729298	0.466041	False
syslog_log2metric_humio_metrics	36.97KiB	0.45	8.07MiB	61.47KiB	0.552486	8.11MiB	302.74KiB	0	-2.28601	0.0227834	False
splunk_hec_indexer_ack_blackhole	26.27KiB	0.11	23.91MiB	973.79KiB	1.38889	23.93MiB	1.29MiB	2.22222	-0.303388	0.76169	False
http_datadog_filter_blackhole	17.29KiB	0.75	2.26MiB	1.07MiB	0.536193	2.28MiB	1.11MiB	1.11111	-0.210298	0.833494	False
http_pipelines_no_grok_blackhole	-28.68KiB	-1.46	1.92MiB	955.84KiB	0.826446	1.9MiB	942.15KiB	1.0929	0.407945	0.683435	False
syslog_log2metric_splunk_hec_metrics	-50.64KiB	-0.52	9.59MiB	67.73KiB	0.630915	9.54MiB	78.48KiB	0	9.07723	1.17017e-18	False
syslog_regex_logs2metric_ddmetrics	-58.15KiB	-0.78	7.29MiB	576.4KiB	0	7.24MiB	491.6KiB	0	1.44415	0.149146	False
splunk_hec_route_s3	-133.41KiB	-1.52	8.58MiB	2.08MiB	0.552486	8.45MiB	2.66MiB	0.269542	0.740072	0.459506	False
syslog_humio_logs	-367.22KiB	-3.73	9.61MiB	169.92KiB	0.552486	9.25MiB	228.6KiB	0	23.5545	9.90014e-87	False
splunk_hec_to_splunk_hec_logs_noack	-374.61KiB	-1.96	18.66MiB	1.38MiB	0	18.3MiB	1.14MiB	2.20994	3.90208	0.000104555	False
splunk_hec_to_splunk_hec_logs_acks	-683.84KiB	-3.67	18.2MiB	1.67MiB	0.275482	17.54MiB	1.4MiB	1.17647	5.74664	1.36398e-08	False
fluent_remap_aws_firehose	-1.03MiB	-1.42	72.66MiB	709.08KiB	0.549451	71.63MiB	598.04KiB	0	21.7771	7.49367e-81	False
datadog_agent_remap_datadog_logs_acks	-1.15MiB	-3.21	35.78MiB	4.2MiB	2.05279	34.63MiB	8.39MiB	6.74487	2.26303	0.0240616	False

Fine details of each soak run.

(experiment, variant)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_remap_aws_firehose', 'baseline')	364	72.66MiB	709.08KiB	69.85MiB	72.6MiB	73.62MiB	73.81MiB	74.14MiB	74.36MiB	-0.0448379
('fluent_remap_aws_firehose', 'comparison')	367	71.63MiB	598.04KiB	70.09MiB	71.68MiB	72.32MiB	72.49MiB	72.84MiB	73.14MiB	-0.291735
('fluent_elasticsearch', 'comparison')	363	65.16MiB	817.36KiB	62.72MiB	65.22MiB	66.18MiB	66.4MiB	66.69MiB	66.85MiB	-0.30431
('fluent_elasticsearch', 'baseline')	362	64.9MiB	609.2KiB	63.13MiB	64.93MiB	65.67MiB	65.85MiB	66.2MiB	66.35MiB	-0.155216
('datadog_agent_remap_datadog_logs', 'comparison')	362	37.99MiB	1.41MiB	35.63MiB	38.06MiB	39.61MiB	39.79MiB	39.96MiB	40.09MiB	-0.0437749
('datadog_agent_remap_datadog_logs', 'baseline')	369	36.08MiB	710.2KiB	34.69MiB	36.08MiB	36.98MiB	37.13MiB	37.49MiB	37.54MiB	0.0794363
('datadog_agent_remap_datadog_logs_acks', 'baseline')	341	35.78MiB	4.2MiB	0B	36.29MiB	37.02MiB	37.14MiB	37.37MiB	37.69MiB	-7.82085
('datadog_agent_remap_datadog_logs_acks', 'comparison')	341	34.63MiB	8.39MiB	0B	36.7MiB	37.85MiB	38.07MiB	38.32MiB	38.67MiB	-3.67539
('datadog_agent_remap_blackhole_acks', 'comparison')	382	30.8MiB	672.54KiB	28.87MiB	30.86MiB	31.61MiB	31.77MiB	31.95MiB	31.99MiB	-0.297282
('datadog_agent_remap_blackhole', 'comparison')	380	30.48MiB	413.22KiB	29.14MiB	30.49MiB	30.99MiB	31.14MiB	31.36MiB	31.72MiB	-0.0951644
('datadog_agent_remap_blackhole_acks', 'baseline')	380	29.7MiB	993.83KiB	27.16MiB	29.6MiB	31.02MiB	31.17MiB	31.56MiB	32.0MiB	0.0810761
('datadog_agent_remap_blackhole', 'baseline')	380	29.21MiB	914.72KiB	27.3MiB	29.23MiB	30.31MiB	30.53MiB	30.87MiB	31.01MiB	-0.0661514
('splunk_hec_indexer_ack_blackhole', 'comparison')	360	23.93MiB	1.29MiB	20.62MiB	23.9MiB	25.45MiB	26.34MiB	27.13MiB	27.47MiB	0.154105
('splunk_hec_indexer_ack_blackhole', 'baseline')	360	23.91MiB	973.79KiB	21.36MiB	23.91MiB	25.12MiB	25.42MiB	26.19MiB	26.85MiB	0.00444764
('http_to_http_noack', 'comparison')	365	21.93MiB	1.66MiB	17.75MiB	21.74MiB	24.12MiB	24.96MiB	26.02MiB	28.21MiB	0.368267
('http_to_http_noack', 'baseline')	363	21.55MiB	1.35MiB	17.28MiB	21.51MiB	23.32MiB	23.86MiB	24.47MiB	25.38MiB	-0.0413309
('splunk_hec_to_splunk_hec_logs_noack', 'baseline')	364	18.66MiB	1.38MiB	14.72MiB	18.59MiB	20.43MiB	20.82MiB	21.82MiB	22.7MiB	0.118394
('splunk_hec_to_splunk_hec_logs_noack', 'comparison')	362	18.3MiB	1.14MiB	15.29MiB	18.19MiB	19.79MiB	20.38MiB	21.33MiB	21.81MiB	0.370802
('splunk_hec_to_splunk_hec_logs_acks', 'baseline')	363	18.2MiB	1.67MiB	13.38MiB	18.24MiB	20.47MiB	20.97MiB	21.55MiB	22.26MiB	-0.0488331
('splunk_hec_to_splunk_hec_logs_acks', 'comparison')	340	17.54MiB	1.4MiB	12.99MiB	17.57MiB	19.25MiB	19.85MiB	20.7MiB	21.6MiB	-0.139517
('syslog_humio_logs', 'baseline')	362	9.61MiB	169.92KiB	9.08MiB	9.67MiB	9.77MiB	9.79MiB	9.82MiB	9.83MiB	-1.01433
('syslog_log2metric_splunk_hec_metrics', 'baseline')	317	9.59MiB	67.73KiB	9.38MiB	9.59MiB	9.67MiB	9.69MiB	9.72MiB	9.72MiB	-0.263313
('syslog_log2metric_splunk_hec_metrics', 'comparison')	370	9.54MiB	78.48KiB	9.36MiB	9.54MiB	9.63MiB	9.65MiB	9.7MiB	9.74MiB	-0.0561045
('syslog_humio_logs', 'comparison')	320	9.25MiB	228.6KiB	8.77MiB	9.14MiB	9.53MiB	9.55MiB	9.59MiB	9.61MiB	0.127758
('syslog_splunk_hec_logs', 'comparison')	363	9.01MiB	179.3KiB	8.63MiB	9.03MiB	9.23MiB	9.25MiB	9.29MiB	9.35MiB	-0.0297332
('syslog_splunk_hec_logs', 'baseline')	362	8.87MiB	200.86KiB	8.53MiB	8.91MiB	9.1MiB	9.12MiB	9.16MiB	9.18MiB	-0.0377452
('splunk_hec_route_s3', 'baseline')	362	8.58MiB	2.08MiB	3.55MiB	8.48MiB	11.35MiB	12.09MiB	13.54MiB	15.52MiB	0.213161
('splunk_hec_route_s3', 'comparison')	371	8.45MiB	2.66MiB	2.86MiB	8.49MiB	12.04MiB	12.87MiB	14.52MiB	15.85MiB	0.131107
('http_to_http_acks', 'comparison')	362	8.19MiB	5.79MiB	0B	7.03MiB	16.86MiB	19.12MiB	21.42MiB	22.08MiB	0.637697
('syslog_log2metric_humio_metrics', 'comparison')	365	8.11MiB	302.74KiB	7.41MiB	8.25MiB	8.42MiB	8.45MiB	8.49MiB	8.5MiB	-0.436352
('syslog_log2metric_humio_metrics', 'baseline')	362	8.07MiB	61.47KiB	7.9MiB	8.08MiB	8.15MiB	8.17MiB	8.21MiB	8.23MiB	-0.130039
('http_to_http_acks', 'baseline')	363	7.47MiB	6.23MiB	0B	4.35MiB	18.11MiB	19.24MiB	21.5MiB	21.75MiB	0.879546
('syslog_regex_logs2metric_ddmetrics', 'baseline')	364	7.29MiB	576.4KiB	5.85MiB	7.4MiB	7.95MiB	8.0MiB	8.03MiB	8.07MiB	-0.644581
('syslog_loki', 'comparison')	363	7.25MiB	250.21KiB	6.49MiB	7.34MiB	7.41MiB	7.44MiB	7.48MiB	7.52MiB	-1.87846
('syslog_regex_logs2metric_ddmetrics', 'comparison')	341	7.24MiB	491.6KiB	6.06MiB	7.4MiB	7.71MiB	7.74MiB	7.76MiB	7.8MiB	-0.939792
('syslog_loki', 'baseline')	342	7.2MiB	182.38KiB	6.81MiB	7.22MiB	7.42MiB	7.44MiB	7.48MiB	7.52MiB	-0.080983
('splunk_transforms_splunk3', 'comparison')	361	4.6MiB	1.52MiB	1.13MiB	4.41MiB	6.67MiB	7.15MiB	8.3MiB	9.12MiB	0.244262
('splunk_transforms_splunk3', 'baseline')	362	4.47MiB	1.65MiB	640.68KiB	4.5MiB	6.64MiB	7.27MiB	8.51MiB	9.83MiB	0.294417
('http_datadog_filter_blackhole', 'comparison')	360	2.28MiB	1.11MiB	156.69KiB	2.26MiB	3.74MiB	4.31MiB	5.07MiB	5.65MiB	0.410573
('http_datadog_filter_blackhole', 'baseline')	373	2.26MiB	1.07MiB	93.77KiB	2.15MiB	3.79MiB	4.21MiB	4.87MiB	5.47MiB	0.451579
('http_pipelines_no_grok_blackhole', 'baseline')	363	1.92MiB	955.84KiB	124.71KiB	1.83MiB	3.22MiB	3.61MiB	4.2MiB	4.95MiB	0.445065
('http_pipelines_no_grok_blackhole', 'comparison')	366	1.9MiB	942.15KiB	0B	1.83MiB	3.08MiB	3.55MiB	4.15MiB	4.51MiB	0.388775
('http_pipelines_blackhole_acks', 'comparison')	367	891.81KiB	820.47KiB	0B	736.04KiB	2.0MiB	2.41MiB	3.07MiB	4.45MiB	1.20385
('http_pipelines_blackhole', 'comparison')	399	884.6KiB	768.38KiB	0B	738.28KiB	1.92MiB	2.27MiB	3.5MiB	4.06MiB	1.17877
('http_pipelines_blackhole_acks', 'baseline')	316	846.08KiB	742.51KiB	0B	618.46KiB	1.91MiB	2.26MiB	2.85MiB	3.13MiB	0.944639
('http_pipelines_blackhole', 'baseline')	376	845.96KiB	706.42KiB	0B	676.5KiB	1.72MiB	2.16MiB	2.96MiB	3.14MiB	1.07078

Dec 29 '21 23:12 github-actions[bot]

Soak Test Results

Baseline: e666b4299a0d3f509d49d9dae80ac160a747d67f Comparison: fa4abbf30f99e9d113d319c6f1cad4f71bcb6c54 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	baseline outlier percentage	comparison mean	comparison stdev	comparison outlier percentage	t-statistic	p-value	erratic
datadog_agent_remap_blackhole	864.34KiB	3.14	26.92MiB	603.9KiB	1.1976	27.77MiB	697.12KiB	0	-17.7516	1.19879e-58	False
datadog_agent_remap_datadog_logs	640.07KiB	1.7	36.68MiB	428.65KiB	0	37.3MiB	1.17MiB	0	-9.5797	6.15971e-20	False
http_to_http_noack	550.19KiB	2.44	22.0MiB	1.82MiB	0.276243	22.54MiB	1.58MiB	0	-4.23968	2.53518e-05	False
syslog_log2metric_splunk_hec_metrics	355.64KiB	3.92	8.86MiB	265.71KiB	6.45161	9.21MiB	223.25KiB	0	-19.4032	1.53948e-67	False
splunk_hec_to_splunk_hec_logs_noack	254.01KiB	1.33	18.62MiB	1.24MiB	0.943396	18.87MiB	1.3MiB	0.273973	-2.55031	0.0109824	False
syslog_log2metric_humio_metrics	252.95KiB	2.97	8.3MiB	226.46KiB	0	8.55MiB	108.25KiB	1.52905	-18.5357	1.43833e-58	False
syslog_loki	238.64KiB	3.35	6.95MiB	102.76KiB	0.275482	7.19MiB	72.66KiB	0.544959	-36.1919	5.06731e-158	True
splunk_transforms_splunk3	171.74KiB	3.75	4.47MiB	1.69MiB	0	4.63MiB	1.47MiB	0.821918	-1.42503	0.154587	False
syslog_regex_logs2metric_ddmetrics	57.75KiB	0.73	7.71MiB	561.83KiB	0	7.76MiB	512.18KiB	0	-1.40595	0.160195	False
http_datadog_filter_blackhole	55.24KiB	2.45	2.2MiB	1.1MiB	0.831025	2.26MiB	1.15MiB	3.59116	-0.642992	0.520434	False
splunk_hec_indexer_ack_blackhole	35.43KiB	0.15	22.79MiB	1.42MiB	0.549451	22.83MiB	1.16MiB	0.552486	-0.359	0.719704	False
syslog_splunk_hec_logs	19.11KiB	0.2	9.55MiB	109.65KiB	4.69208	9.57MiB	100.69KiB	1.10497	-2.4027	0.016539	False
http_pipelines_no_grok_blackhole	-7.73KiB	-0.37	2.02MiB	933.42KiB	1.10803	2.02MiB	1008.59KiB	0.275482	0.107061	0.91477	False
http_pipelines_blackhole_acks	-35.53KiB	-4.01	885.68KiB	704.23KiB	1.89873	850.15KiB	698.04KiB	2.77008	0.657649	0.510992	False
http_pipelines_blackhole	-51.89KiB	-5.84	889.04KiB	743.78KiB	2.47934	837.15KiB	712.88KiB	2.22222	0.957687	0.338542	False
datadog_agent_remap_blackhole_acks	-214.31KiB	-0.71	29.62MiB	1.02MiB	0	29.42MiB	1.92MiB	0	1.78512	0.0748546	False
splunk_hec_to_splunk_hec_logs_acks	-426.33KiB	-2.21	18.86MiB	1.45MiB	0.828729	18.44MiB	1.21MiB	0.879765	4.1387	3.92403e-05	False
splunk_hec_route_s3	-433.65KiB	-4.97	8.52MiB	2.38MiB	1.08401	8.1MiB	2.02MiB	1.46628	2.56334	0.0105739	False
syslog_humio_logs	-501.35KiB	-5.03	9.74MiB	67.3KiB	1.65289	9.25MiB	469.65KiB	0	19.5811	1.98065e-58	False
fluent_remap_aws_firehose	-580.98KiB	-0.78	72.6MiB	556.01KiB	1.46628	72.04MiB	906.36KiB	2.47253	10.3295	3.72416e-23	False
fluent_elasticsearch	-1.48MiB	-2.14	69.31MiB	591.92KiB	2.63158	67.83MiB	1.67MiB	0	15.6212	7.67737e-44	False
datadog_agent_remap_datadog_logs_acks	-1.65MiB	-4.29	38.4MiB	796.6KiB	0	36.75MiB	6.07MiB	4.69613	5.11941	4.92137e-07	False
http_to_http_acks	-3.85MiB	-36.11	10.66MiB	7.63MiB	0	6.81MiB	4.91MiB	1.9337	7.90293	1.3959e-14	True

Fine details of each soak run.

(experiment, variant)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_remap_aws_firehose', 'baseline')	341	72.6MiB	556.01KiB	70.88MiB	72.59MiB	73.26MiB	73.47MiB	73.88MiB	74.19MiB	-0.132981
('fluent_remap_aws_firehose', 'comparison')	364	72.04MiB	906.36KiB	63.97MiB	72.1MiB	72.8MiB	73.04MiB	73.48MiB	73.95MiB	-4.15652
('fluent_elasticsearch', 'baseline')	342	69.31MiB	591.92KiB	66.9MiB	69.33MiB	69.99MiB	70.16MiB	70.47MiB	70.88MiB	-0.654089
('fluent_elasticsearch', 'comparison')	346	67.83MiB	1.67MiB	63.51MiB	67.34MiB	69.9MiB	70.02MiB	70.33MiB	70.86MiB	-0.0301297
('datadog_agent_remap_datadog_logs_acks', 'baseline')	364	38.4MiB	796.6KiB	36.47MiB	38.44MiB	39.4MiB	39.56MiB	39.8MiB	40.0MiB	-0.150964
('datadog_agent_remap_datadog_logs', 'comparison')	363	37.3MiB	1.17MiB	35.53MiB	36.93MiB	38.7MiB	38.87MiB	39.12MiB	39.26MiB	0.0529384
('datadog_agent_remap_datadog_logs_acks', 'comparison')	362	36.75MiB	6.07MiB	0B	37.55MiB	39.22MiB	39.39MiB	39.71MiB	40.22MiB	-5.41727
('datadog_agent_remap_datadog_logs', 'baseline')	342	36.68MiB	428.65KiB	35.49MiB	36.68MiB	37.23MiB	37.33MiB	37.61MiB	37.72MiB	-0.00913488
('datadog_agent_remap_blackhole_acks', 'baseline')	380	29.62MiB	1.02MiB	27.3MiB	29.6MiB	30.91MiB	31.09MiB	31.35MiB	31.79MiB	-0.0612289
('datadog_agent_remap_blackhole_acks', 'comparison')	336	29.42MiB	1.92MiB	26.24MiB	28.64MiB	31.78MiB	31.94MiB	32.11MiB	32.36MiB	0.115405
('datadog_agent_remap_blackhole', 'comparison')	380	27.77MiB	697.12KiB	25.95MiB	27.72MiB	28.73MiB	28.88MiB	29.31MiB	29.6MiB	0.215358
('datadog_agent_remap_blackhole', 'baseline')	334	26.92MiB	603.9KiB	25.05MiB	26.91MiB	27.71MiB	27.96MiB	28.35MiB	28.79MiB	0.0874228
('splunk_hec_indexer_ack_blackhole', 'comparison')	362	22.83MiB	1.16MiB	19.9MiB	22.82MiB	24.34MiB	24.67MiB	25.66MiB	26.22MiB	0.0789924
('splunk_hec_indexer_ack_blackhole', 'baseline')	364	22.79MiB	1.42MiB	19.07MiB	22.74MiB	24.54MiB	25.32MiB	26.44MiB	26.79MiB	0.299451
('http_to_http_noack', 'comparison')	361	22.54MiB	1.58MiB	19.15MiB	22.58MiB	24.6MiB	25.05MiB	25.87MiB	26.6MiB	-0.00165675
('http_to_http_noack', 'baseline')	362	22.0MiB	1.82MiB	17.63MiB	22.01MiB	24.37MiB	24.93MiB	26.36MiB	27.52MiB	0.17095
('splunk_hec_to_splunk_hec_logs_noack', 'comparison')	365	18.87MiB	1.3MiB	14.93MiB	18.71MiB	20.6MiB	21.15MiB	21.67MiB	21.93MiB	0.0786927
('splunk_hec_to_splunk_hec_logs_acks', 'baseline')	362	18.86MiB	1.45MiB	14.18MiB	18.95MiB	20.59MiB	21.09MiB	21.71MiB	23.63MiB	-0.26792
('splunk_hec_to_splunk_hec_logs_noack', 'baseline')	318	18.62MiB	1.24MiB	14.87MiB	18.55MiB	20.29MiB	20.77MiB	21.35MiB	21.96MiB	0.103764
('splunk_hec_to_splunk_hec_logs_acks', 'comparison')	341	18.44MiB	1.21MiB	14.51MiB	18.47MiB	19.96MiB	20.2MiB	21.01MiB	21.51MiB	-0.321195
('http_to_http_acks', 'baseline')	341	10.66MiB	7.63MiB	488.08KiB	9.2MiB	21.42MiB	21.72MiB	22.97MiB	23.8MiB	0.242098
('syslog_humio_logs', 'baseline')	363	9.74MiB	67.3KiB	9.54MiB	9.74MiB	9.82MiB	9.85MiB	9.89MiB	9.93MiB	-0.248265
('syslog_splunk_hec_logs', 'comparison')	362	9.57MiB	100.69KiB	9.24MiB	9.57MiB	9.69MiB	9.72MiB	9.76MiB	9.94MiB	-0.228549
('syslog_splunk_hec_logs', 'baseline')	341	9.55MiB	109.65KiB	9.15MiB	9.56MiB	9.68MiB	9.71MiB	9.76MiB	9.84MiB	-0.668284
('syslog_humio_logs', 'comparison')	343	9.25MiB	469.65KiB	8.58MiB	9.57MiB	9.73MiB	9.76MiB	9.79MiB	9.86MiB	-0.114944
('syslog_log2metric_splunk_hec_metrics', 'comparison')	341	9.21MiB	223.25KiB	8.88MiB	9.09MiB	9.47MiB	9.49MiB	9.52MiB	9.52MiB	0.0887955
('syslog_log2metric_splunk_hec_metrics', 'baseline')	372	8.86MiB	265.71KiB	7.97MiB	8.84MiB	9.12MiB	9.15MiB	9.17MiB	9.19MiB	-1.49881
('syslog_log2metric_humio_metrics', 'comparison')	327	8.55MiB	108.25KiB	8.34MiB	8.55MiB	8.69MiB	8.74MiB	8.84MiB	8.92MiB	0.53613
('splunk_hec_route_s3', 'baseline')	369	8.52MiB	2.38MiB	2.74MiB	8.4MiB	11.6MiB	12.38MiB	14.55MiB	17.4MiB	0.395903
('syslog_log2metric_humio_metrics', 'baseline')	341	8.3MiB	226.46KiB	7.96MiB	8.19MiB	8.58MiB	8.6MiB	8.63MiB	8.67MiB	0.128621
('splunk_hec_route_s3', 'comparison')	341	8.1MiB	2.02MiB	2.92MiB	7.93MiB	10.73MiB	11.59MiB	13.36MiB	14.01MiB	0.301818
('syslog_regex_logs2metric_ddmetrics', 'comparison')	319	7.76MiB	512.18KiB	6.73MiB	7.95MiB	8.24MiB	8.26MiB	8.31MiB	8.33MiB	-0.677671
('syslog_regex_logs2metric_ddmetrics', 'baseline')	365	7.71MiB	561.83KiB	6.55MiB	7.94MiB	8.28MiB	8.31MiB	8.36MiB	8.38MiB	-0.701664
('syslog_loki', 'comparison')	367	7.19MiB	72.66KiB	6.96MiB	7.2MiB	7.27MiB	7.29MiB	7.32MiB	7.35MiB	-0.442319
('syslog_loki', 'baseline')	363	6.95MiB	102.76KiB	6.63MiB	6.96MiB	7.08MiB	7.1MiB	7.13MiB	7.18MiB	-0.392837
('http_to_http_acks', 'comparison')	362	6.81MiB	4.91MiB	1.46MiB	5.06MiB	14.73MiB	17.03MiB	20.17MiB	21.27MiB	1.06715
('splunk_transforms_splunk3', 'comparison')	365	4.63MiB	1.47MiB	1.19MiB	4.59MiB	6.53MiB	7.06MiB	8.25MiB	11.17MiB	0.440524
('splunk_transforms_splunk3', 'baseline')	362	4.47MiB	1.69MiB	732.2KiB	4.37MiB	6.73MiB	7.18MiB	8.58MiB	8.91MiB	0.206545
('http_datadog_filter_blackhole', 'comparison')	362	2.26MiB	1.15MiB	278.01KiB	2.1MiB	3.68MiB	4.48MiB	5.7MiB	6.6MiB	0.878036
('http_datadog_filter_blackhole', 'baseline')	361	2.2MiB	1.1MiB	0B	2.16MiB	3.65MiB	4.25MiB	5.02MiB	5.46MiB	0.497084
('http_pipelines_no_grok_blackhole', 'baseline')	361	2.02MiB	933.42KiB	156.69KiB	2.01MiB	3.19MiB	3.5MiB	4.46MiB	5.73MiB	0.417742
('http_pipelines_no_grok_blackhole', 'comparison')	363	2.02MiB	1008.59KiB	246.03KiB	1.98MiB	3.35MiB	3.74MiB	4.41MiB	4.92MiB	0.320637
('http_pipelines_blackhole', 'baseline')	363	889.04KiB	743.78KiB	0B	708.48KiB	1.91MiB	2.36MiB	2.92MiB	3.56MiB	1.10255
('http_pipelines_blackhole_acks', 'baseline')	316	885.68KiB	704.23KiB	0B	782.92KiB	1.78MiB	2.0MiB	3.04MiB	4.35MiB	1.08851
('http_pipelines_blackhole_acks', 'comparison')	361	850.15KiB	698.04KiB	0B	678.68KiB	1.7MiB	2.18MiB	2.83MiB	3.2MiB	1.02304
('http_pipelines_blackhole', 'comparison')	360	837.15KiB	712.88KiB	0B	646.13KiB	1.8MiB	2.15MiB	2.89MiB	3.47MiB	1.08745

Dec 30 '21 17:12 github-actions[bot]

Soak Test Results

Baseline: cdfa82d32be84c4b140f2d2971cee7f9c6610878 Comparison: 30ac12334eeaf97d0b3121f9ef95b27f5a211485 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	comparison mean	comparison stdev	p-value	erratic
datadog_agent_remap_datadog_logs	1.87MiB	5.08	36.82MiB	719.02KiB	38.69MiB	399.88KiB	9.36589e-307	False
datadog_agent_remap_blackhole	1.06MiB	3.95	26.92MiB	848.32KiB	27.98MiB	1.47MiB	3.06793e-48	False
splunk_hec_indexer_ack_blackhole	1.02MiB	5.23	19.53MiB	1008.97KiB	20.55MiB	906.81KiB	1.12506e-69	False
syslog_log2metric_splunk_hec_metrics	572.32KiB	6.16	9.07MiB	269.49KiB	9.63MiB	59.08KiB	1.10569e-229	False
syslog_regex_logs2metric_ddmetrics	561.46KiB	7.63	7.19MiB	410.49KiB	7.73MiB	322.25KiB	8.7582e-120	False
datadog_agent_remap_blackhole_acks	502.71KiB	1.74	28.19MiB	711.19KiB	28.68MiB	1.19MiB	1.19552e-17	False
fluent_remap_aws_firehose	378.38KiB	0.51	72.65MiB	1.88MiB	73.02MiB	1.58MiB	0.000219985	False
syslog_humio_logs	84.03KiB	0.85	9.7MiB	850.87KiB	9.78MiB	100.07KiB	0.0320766	False
http_pipelines_blackhole_acks	40.59KiB	4.86	834.97KiB	521.07KiB	875.56KiB	534.55KiB	0.184269	False
http_datadog_filter_blackhole	10.55KiB	0.47	2.17MiB	839.41KiB	2.18MiB	841.05KiB	0.827033	False
http_pipelines_no_grok_blackhole	-46.65KiB	-2.3	1.98MiB	831.59KiB	1.93MiB	827.73KiB	0.329868	False
http_pipelines_blackhole	-59.3KiB	-6.98	849.56KiB	518.64KiB	790.26KiB	502.18KiB	0.0456137	False
syslog_splunk_hec_logs	-68.67KiB	-0.71	9.46MiB	207.77KiB	9.4MiB	318.8KiB	1.10717e-05	False
http_to_http_acks	-81.57KiB	-0.83	9.58MiB	4.34MiB	9.5MiB	4.22MiB	0.747361	False
splunk_hec_route_s3	-106.33KiB	-1.21	8.56MiB	1.95MiB	8.46MiB	1.98MiB	0.362094	False
splunk_transforms_splunk3	-109.04KiB	-2.01	5.29MiB	1.47MiB	5.19MiB	1.51MiB	0.212919	False
syslog_log2metric_humio_metrics	-224.44KiB	-2.73	8.03MiB	88.71KiB	7.81MiB	50.64KiB	8.53279e-294	False
syslog_loki	-252.22KiB	-3.55	6.95MiB	207.21KiB	6.7MiB	70.87KiB	1.79838e-120	False
splunk_hec_to_splunk_hec_logs_noack	-402.31KiB	-2.13	18.45MiB	886.1KiB	18.05MiB	986.64KiB	2.04895e-13	False
http_to_http_noack	-507.85KiB	-2.3	21.53MiB	898.02KiB	21.04MiB	1.13MiB	6.9092e-17	False
splunk_hec_to_splunk_hec_logs_acks	-664.9KiB	-3.56	18.23MiB	1015.08KiB	17.58MiB	1.01MiB	9.27531e-28	False
datadog_agent_remap_datadog_logs_acks	-741.6KiB	-1.97	36.81MiB	1.25MiB	36.09MiB	1.11MiB	1.99613e-25	False
fluent_elasticsearch	-1.19MiB	-1.7	69.94MiB	1.91MiB	68.75MiB	1.97MiB	4.68216e-25	False

Fine details of each soak run.

(experiment, variant, run_id)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_remap_aws_firehose', 'baseline', '834543d3-3f7b-4680-be0a-fe59cbb48387')	201	74.82MiB	364.11KiB	73.9MiB	74.83MiB	75.24MiB	75.41MiB	75.48MiB	75.56MiB	-0.209551
('fluent_remap_aws_firehose', 'comparison', 'a7bd1eb5-d18d-4d5c-ae76-9441df84dcd5')	202	74.76MiB	357.61KiB	73.84MiB	74.78MiB	75.2MiB	75.32MiB	75.45MiB	75.61MiB	-0.159553
('fluent_remap_aws_firehose', 'comparison', '316d4de1-4707-4b51-98d9-74042150b933')	201	73.29MiB	383.31KiB	72.2MiB	73.32MiB	73.74MiB	73.83MiB	73.95MiB	74.03MiB	-0.518293
('fluent_remap_aws_firehose', 'baseline', '369682b8-7e76-42ad-b8a4-38b3b84f8988')	203	72.84MiB	514.66KiB	71.64MiB	72.8MiB	73.49MiB	73.7MiB	74.0MiB	74.22MiB	0.145839
('fluent_elasticsearch', 'comparison', '4e6fc6b3-39ed-40d6-964e-99d2671b1dc4')	199	71.49MiB	372.87KiB	70.7MiB	71.47MiB	71.92MiB	72.15MiB	72.31MiB	72.45MiB	0.214194
('fluent_elasticsearch', 'baseline', 'b98e6be9-795f-4bf0-aef3-b695e1f58c5b')	199	71.27MiB	436.45KiB	70.26MiB	71.26MiB	71.84MiB	71.97MiB	72.13MiB	72.27MiB	-0.0310071
('fluent_elasticsearch', 'baseline', '0f458240-9f4c-447e-a885-78951f87d06e')	199	71.23MiB	361.02KiB	70.46MiB	71.24MiB	71.72MiB	71.79MiB	72.02MiB	72.27MiB	0.0367528
('fluent_remap_aws_firehose', 'comparison', '09e1de90-f0bf-4dda-95f9-fddbc164c157')	203	71.04MiB	378.99KiB	70.2MiB	71.04MiB	71.47MiB	71.6MiB	71.89MiB	72.15MiB	0.0766328
('fluent_remap_aws_firehose', 'baseline', 'a5eaf7a5-645e-4057-a92d-e41f5269e6eb')	203	70.32MiB	295.84KiB	69.48MiB	70.32MiB	70.69MiB	70.74MiB	70.9MiB	70.93MiB	-0.205266
('fluent_elasticsearch', 'comparison', 'aed1ddd0-dd5c-46c6-99f8-c718bbf3b73b')	199	67.53MiB	341.37KiB	66.34MiB	67.54MiB	67.93MiB	68.06MiB	68.15MiB	68.32MiB	-0.334959
('fluent_elasticsearch', 'baseline', '814346e6-0b91-483d-8046-52a6c825889a')	198	67.3MiB	448.69KiB	66.11MiB	67.3MiB	67.89MiB	68.02MiB	68.29MiB	68.33MiB	-0.0701779
('fluent_elasticsearch', 'comparison', '02c74d77-7e1d-4565-9173-ad363685af9b')	199	67.24MiB	310.57KiB	66.5MiB	67.23MiB	67.63MiB	67.73MiB	67.85MiB	68.02MiB	-0.0621097
('datadog_agent_remap_datadog_logs', 'comparison', '02516e26-d1c7-4a2e-a67c-298770d8d4f4')	200	38.97MiB	252.83KiB	38.23MiB	38.96MiB	39.27MiB	39.38MiB	39.62MiB	39.73MiB	0.140332
('datadog_agent_remap_datadog_logs', 'comparison', 'ba2b5322-ff20-40c7-9f78-2cfaaa30fdf6')	200	38.81MiB	304.39KiB	37.9MiB	38.81MiB	39.2MiB	39.29MiB	39.41MiB	39.44MiB	-0.145773
('datadog_agent_remap_datadog_logs_acks', 'baseline', '11ed4eb6-0a2d-4798-a2d6-4bddbb9bbaf2')	199	38.42MiB	542.75KiB	31.93MiB	38.46MiB	38.79MiB	38.87MiB	39.06MiB	39.23MiB	-9.36676
('datadog_agent_remap_datadog_logs', 'comparison', 'e3514bb7-6136-485f-a994-b715caa57ecf')	199	38.29MiB	248.21KiB	37.5MiB	38.3MiB	38.59MiB	38.65MiB	38.85MiB	39.21MiB	0.0248801
('datadog_agent_remap_datadog_logs', 'baseline', 'c8cd9e6a-5cfc-40ef-a3d7-9d967fec91e0')	200	37.74MiB	234.96KiB	37.22MiB	37.74MiB	38.03MiB	38.12MiB	38.3MiB	38.59MiB	0.325009
('datadog_agent_remap_datadog_logs_acks', 'comparison', '3e4f9b59-f0ee-4b16-a6f4-f86393e9c38c')	202	37.1MiB	245.61KiB	36.41MiB	37.11MiB	37.38MiB	37.46MiB	37.74MiB	37.76MiB	-0.0332074
('datadog_agent_remap_datadog_logs_acks', 'comparison', '74e4a3ba-a340-4b9a-ae5b-910d33febd3a')	202	36.57MiB	245.63KiB	35.68MiB	36.58MiB	36.89MiB	36.99MiB	37.08MiB	37.17MiB	-0.115523
('datadog_agent_remap_datadog_logs', 'baseline', 'd026c58b-1ca5-4437-8cf0-86ef7f90dcd6')	200	36.52MiB	210.93KiB	36.04MiB	36.5MiB	36.81MiB	36.88MiB	36.99MiB	37.02MiB	0.166704
('datadog_agent_remap_datadog_logs', 'baseline', '0f266617-77bc-4530-8970-623cea9297ab')	200	36.2MiB	238.79KiB	35.37MiB	36.2MiB	36.48MiB	36.56MiB	36.73MiB	36.79MiB	-0.348524
('datadog_agent_remap_datadog_logs_acks', 'baseline', 'af16effc-913e-4047-95f1-70bda09935c6')	203	36.08MiB	688.49KiB	34.01MiB	36.31MiB	36.68MiB	36.75MiB	36.91MiB	37.01MiB	-1.43247
('datadog_agent_remap_datadog_logs_acks', 'baseline', '4ca521b1-5757-4320-b84d-d8e22833d3e2')	199	35.95MiB	276.52KiB	35.22MiB	35.95MiB	36.28MiB	36.35MiB	36.57MiB	37.08MiB	0.187938
('datadog_agent_remap_datadog_logs_acks', 'comparison', '9559180f-6c15-4d0a-8b2a-c6cc4037e4eb')	202	34.6MiB	262.33KiB	33.98MiB	34.61MiB	34.94MiB	35.06MiB	35.12MiB	35.17MiB	-0.114071
('datadog_agent_remap_blackhole_acks', 'comparison', '92f12170-748f-4e57-a2b2-674e8968ece5')	200	29.81MiB	461.35KiB	28.37MiB	29.82MiB	30.42MiB	30.57MiB	30.78MiB	30.88MiB	-0.11218
('datadog_agent_remap_blackhole', 'comparison', '2550d85f-cc2d-4730-bafb-a406cfb9083a')	201	29.5MiB	404.18KiB	28.32MiB	29.5MiB	30.01MiB	30.14MiB	30.47MiB	30.56MiB	0.0247834
('datadog_agent_remap_blackhole_acks', 'baseline', 'd487a4ca-81f6-44f6-9535-d685c297bf19')	200	28.93MiB	480.33KiB	27.69MiB	28.96MiB	29.49MiB	29.68MiB	30.04MiB	30.32MiB	0.0965487
('datadog_agent_remap_blackhole_acks', 'comparison', '62735542-74ef-45ca-86f6-de2b8c25e728')	200	28.91MiB	395.81KiB	27.76MiB	28.93MiB	29.38MiB	29.49MiB	29.93MiB	30.05MiB	0.0483137
('datadog_agent_remap_blackhole', 'comparison', '657413ab-14db-47f5-8c68-53f9076316e2')	201	28.21MiB	793.47KiB	24.84MiB	28.2MiB	29.14MiB	29.4MiB	29.59MiB	29.92MiB	-0.541176
('datadog_agent_remap_blackhole_acks', 'baseline', 'aeecb71b-94ef-412f-be39-09a59f9f329c')	200	27.86MiB	496.24KiB	26.51MiB	27.9MiB	28.51MiB	28.64MiB	29.08MiB	29.21MiB	0.0131347
('datadog_agent_remap_blackhole_acks', 'baseline', 'b9678e9e-c5db-4fda-a44c-1b42052cb774')	200	27.78MiB	426.0KiB	26.61MiB	27.8MiB	28.28MiB	28.46MiB	28.72MiB	28.93MiB	-0.0492221
('datadog_agent_remap_blackhole', 'baseline', '318db798-b21d-42b2-9fb8-2747ae815341')	202	27.48MiB	451.48KiB	26.21MiB	27.51MiB	28.04MiB	28.15MiB	28.42MiB	28.81MiB	-0.186135
('datadog_agent_remap_blackhole_acks', 'comparison', '3f749aee-caee-43cb-9582-3c386afec0d8')	199	27.31MiB	849.57KiB	24.87MiB	27.43MiB	28.39MiB	28.53MiB	28.73MiB	29.19MiB	-0.504603
('datadog_agent_remap_blackhole', 'baseline', '7e1638f5-ab19-4d7f-b329-21e29cc155ed')	201	26.8MiB	794.62KiB	25.05MiB	26.78MiB	27.86MiB	28.23MiB	28.54MiB	28.73MiB	0.215692
('datadog_agent_remap_blackhole', 'baseline', '9080b526-4244-4b80-9c1d-6cf7e97a641c')	202	26.48MiB	886.01KiB	23.78MiB	26.53MiB	27.5MiB	27.77MiB	28.24MiB	28.72MiB	-0.298021
('datadog_agent_remap_blackhole', 'comparison', '0ecda5a6-4dca-4b35-a40b-69efd5eb6bfd')	201	26.25MiB	617.53KiB	24.52MiB	26.19MiB	27.0MiB	27.22MiB	27.71MiB	28.05MiB	0.0668684
('http_to_http_noack', 'baseline', '5d9315c3-51e1-4be6-b066-a8bb261ad348')	199	21.94MiB	944.75KiB	17.87MiB	21.85MiB	23.1MiB	23.39MiB	24.12MiB	24.83MiB	-0.0336196
('http_to_http_noack', 'comparison', 'd49b393a-d2a9-4e28-8fec-235d2a02b2ec')	199	21.81MiB	1.2MiB	18.69MiB	22.0MiB	23.13MiB	23.58MiB	23.97MiB	24.68MiB	-0.433148
('http_to_http_noack', 'baseline', 'bf2935f7-14b1-4c56-b416-3e4ae92158bb')	199	21.53MiB	730.14KiB	20.06MiB	21.44MiB	22.6MiB	22.73MiB	22.95MiB	23.22MiB	0.0753081
('splunk_hec_indexer_ack_blackhole', 'comparison', '0516fedf-20a7-4ff2-8a7e-dfbfcb160c4b')	200	21.16MiB	748.95KiB	19.0MiB	21.13MiB	22.14MiB	22.34MiB	22.81MiB	23.12MiB	0.0996329
('http_to_http_noack', 'baseline', 'dbaa9d39-db2f-4617-b461-51d217dccb38')	199	21.13MiB	809.8KiB	19.11MiB	21.12MiB	22.22MiB	22.45MiB	22.68MiB	22.94MiB	-0.039078
('http_to_http_noack', 'comparison', '47106e70-c03e-4572-9bb5-906bc42729ff')	199	20.8MiB	913.67KiB	18.09MiB	20.72MiB	22.03MiB	22.36MiB	22.73MiB	23.31MiB	0.233028
('http_to_http_noack', 'comparison', 'dcf68bdd-8168-455d-8723-5dd406059e9f')	200	20.51MiB	840.98KiB	18.29MiB	20.47MiB	21.47MiB	21.79MiB	22.63MiB	23.16MiB	0.158326
('splunk_hec_indexer_ack_blackhole', 'comparison', '2b9aa090-32c1-4c6b-853f-f52566da4660')	200	20.29MiB	797.16KiB	18.01MiB	20.33MiB	21.37MiB	21.63MiB	21.92MiB	22.05MiB	0.0390468
('splunk_hec_indexer_ack_blackhole', 'comparison', 'daa30097-2da2-4329-9ddc-8bacc7f6c9fe')	200	20.2MiB	828.72KiB	18.3MiB	20.14MiB	21.29MiB	21.65MiB	22.16MiB	22.53MiB	0.309992
('splunk_hec_indexer_ack_blackhole', 'baseline', '0653dce8-f5bc-4c25-b1af-467fdeb9fc6b')	200	20.08MiB	760.44KiB	18.28MiB	20.09MiB	20.93MiB	21.15MiB	21.83MiB	22.36MiB	-0.0185275
('splunk_hec_indexer_ack_blackhole', 'baseline', 'cf5b274e-a170-48a9-97a3-b68637332d7d')	200	19.81MiB	759.17KiB	17.76MiB	19.79MiB	20.73MiB	21.02MiB	21.53MiB	21.65MiB	-0.0801263
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', '35c3ffd5-17a9-440a-94ae-93e050832be6')	199	19.0MiB	707.68KiB	17.04MiB	19.07MiB	19.82MiB	20.09MiB	20.4MiB	20.93MiB	-0.241285
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', 'f775b7ef-a996-4ca9-a465-6793723d6977')	200	18.91MiB	709.43KiB	16.98MiB	18.99MiB	19.7MiB	20.04MiB	20.29MiB	20.46MiB	-0.33433
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', '1afa0fe9-ec72-4e68-bef1-c1f54b982c81')	200	18.71MiB	721.12KiB	16.75MiB	18.8MiB	19.56MiB	19.73MiB	20.07MiB	20.26MiB	-0.375707
('splunk_hec_indexer_ack_blackhole', 'baseline', '24ae9e00-e536-40c1-acc4-2f7777d35a8f')	200	18.69MiB	875.36KiB	16.45MiB	18.7MiB	19.88MiB	20.05MiB	20.28MiB	20.65MiB	-0.190513
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', '4d3c9b25-1ab5-477e-866b-15cea24c5f50')	199	18.57MiB	871.18KiB	15.61MiB	18.59MiB	19.59MiB	19.88MiB	20.16MiB	20.38MiB	-0.595253
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', 'd61f7ac1-5874-42ce-8249-be44f687ddba')	200	18.56MiB	958.37KiB	15.94MiB	18.7MiB	19.67MiB	19.83MiB	20.25MiB	20.26MiB	-0.572869
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', 'f9e769be-61bf-4693-8103-374535532971')	198	18.2MiB	975.98KiB	15.49MiB	18.34MiB	19.37MiB	19.63MiB	19.94MiB	20.53MiB	-0.420752
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', '9afd0c3a-7dbe-40be-bcbb-b563b90c5c53')	204	17.73MiB	699.71KiB	16.09MiB	17.65MiB	18.66MiB	18.78MiB	19.07MiB	19.4MiB	-0.00414702
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', 'bd9d0293-adea-499d-b49a-70df933d403b')	199	17.69MiB	753.32KiB	15.66MiB	17.73MiB	18.61MiB	18.74MiB	19.33MiB	19.86MiB	-0.108159
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', '083a0ea3-241a-4123-a495-24870a24eeec')	199	17.57MiB	852.17KiB	15.81MiB	17.65MiB	18.63MiB	18.91MiB	19.34MiB	19.41MiB	-0.0370317
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', '24accfea-674a-40c8-ba8b-3b02e378a3c9')	199	17.46MiB	633.02KiB	15.82MiB	17.48MiB	18.3MiB	18.38MiB	18.68MiB	18.91MiB	-0.123311
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', '6f369f7d-72a2-4ca3-a1a6-9d460cb0c7a4')	198	17.29MiB	829.34KiB	14.91MiB	17.27MiB	18.34MiB	18.49MiB	19.19MiB	19.74MiB	0.0430906
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', '1ee5cc4f-f9ca-40fb-a8fa-abb578500c27')	198	17.25MiB	983.02KiB	14.85MiB	17.3MiB	18.5MiB	18.76MiB	19.13MiB	19.28MiB	-0.192442
('http_to_http_acks', 'baseline', '096b062a-9aff-4a96-90f8-2ab77ca643c6')	200	10.26MiB	4.92MiB	2.01MiB	10.02MiB	17.8MiB	19.19MiB	20.18MiB	20.36MiB	0.329197
('http_to_http_acks', 'comparison', '42338066-581c-4783-9fca-83f3e2a851e3')	199	10.22MiB	4.4MiB	2.43MiB	10.11MiB	16.76MiB	19.04MiB	20.76MiB	21.36MiB	0.550804
('syslog_humio_logs', 'comparison', 'b1183e22-7532-4b10-a521-ad5987ef6288')	199	9.85MiB	31.89KiB	9.78MiB	9.85MiB	9.89MiB	9.9MiB	9.92MiB	9.95MiB	0.146522
('syslog_splunk_hec_logs', 'comparison', '6bbfda27-5347-4cd9-8bd7-e4e1674cfba7')	200	9.83MiB	45.17KiB	9.72MiB	9.84MiB	9.89MiB	9.9MiB	9.93MiB	9.94MiB	-0.184476
('syslog_humio_logs', 'comparison', '748af768-ea5f-45b0-92e0-572043c6c7c5')	199	9.83MiB	47.53KiB	9.73MiB	9.83MiB	9.89MiB	9.9MiB	9.92MiB	9.94MiB	0.052766
('syslog_humio_logs', 'baseline', '6cc8b4e1-c059-4cca-ac4f-7e731378b11e')	202	9.82MiB	32.26KiB	9.71MiB	9.82MiB	9.86MiB	9.87MiB	9.88MiB	9.91MiB	-0.175319
('http_to_http_acks', 'baseline', '721c5542-10ea-48f4-91ea-77d3d3f30c59')	200	9.74MiB	4.3MiB	1.34MiB	9.51MiB	16.44MiB	17.76MiB	19.56MiB	20.2MiB	0.430809
('syslog_humio_logs', 'baseline', '30314b3c-52ad-4b7a-926a-4088981ac29e')	202	9.72MiB	33.7KiB	9.62MiB	9.72MiB	9.76MiB	9.77MiB	9.78MiB	9.8MiB	-0.195406
('syslog_log2metric_splunk_hec_metrics', 'comparison', '26c325d0-e027-4712-b6ce-84ccb742f02b')	200	9.68MiB	28.57KiB	9.61MiB	9.67MiB	9.71MiB	9.72MiB	9.73MiB	9.76MiB	-0.135705
('syslog_splunk_hec_logs', 'baseline', '608f73bd-edda-4392-83a7-27d39a28721c')	199	9.67MiB	45.21KiB	9.56MiB	9.67MiB	9.73MiB	9.74MiB	9.77MiB	9.78MiB	-0.137411
('syslog_log2metric_splunk_hec_metrics', 'comparison', 'f3fb20ce-307b-4d2c-8e24-e4a82e21119f')	200	9.66MiB	27.95KiB	9.6MiB	9.66MiB	9.69MiB	9.71MiB	9.72MiB	9.73MiB	0.115879
('syslog_humio_logs', 'comparison', '2324be3a-cd44-4d9d-9d52-c11a20a70b9e')	199	9.65MiB	47.71KiB	9.53MiB	9.66MiB	9.72MiB	9.73MiB	9.74MiB	9.74MiB	-0.0905918
('syslog_log2metric_splunk_hec_metrics', 'comparison', '5a36ed50-8404-42f0-a0f8-0e7c56e12978')	200	9.57MiB	38.72KiB	9.45MiB	9.56MiB	9.61MiB	9.62MiB	9.66MiB	9.71MiB	0.294776
('syslog_splunk_hec_logs', 'baseline', 'd9ee9a00-bc7c-4d78-ab3f-9d9e0633a52c')	199	9.51MiB	98.45KiB	9.28MiB	9.5MiB	9.66MiB	9.68MiB	9.74MiB	9.77MiB	0.429977
('http_to_http_acks', 'comparison', '56554c92-b206-4f4c-a844-345807958df6')	199	9.49MiB	4.38MiB	1.85MiB	9.49MiB	16.1MiB	18.62MiB	20.47MiB	21.06MiB	0.691281
('syslog_log2metric_splunk_hec_metrics', 'baseline', '90097f8c-7bc6-4134-b03c-ea145bdfced3')	200	9.35MiB	42.84KiB	9.24MiB	9.35MiB	9.4MiB	9.42MiB	9.43MiB	9.46MiB	-0.184891
('syslog_humio_logs', 'baseline', '47713169-51d5-46ee-b6c4-426f59edfdfa')	75	9.31MiB	2.06MiB	49.53KiB	9.82MiB	9.86MiB	9.87MiB	9.89MiB	9.9MiB	-4.11351
('syslog_splunk_hec_logs', 'baseline', 'e0a732d0-053f-40cc-934d-766b42ef4fe6')	199	9.21MiB	62.57KiB	9.06MiB	9.22MiB	9.28MiB	9.29MiB	9.32MiB	9.33MiB	-0.531037
('syslog_splunk_hec_logs', 'comparison', 'd5ca11b5-d441-47ec-b6fb-aacd6e32de4a')	200	9.2MiB	36.6KiB	9.07MiB	9.2MiB	9.24MiB	9.25MiB	9.27MiB	9.28MiB	-0.444599
('syslog_splunk_hec_logs', 'comparison', 'ed3feb18-0d1a-4150-8810-6cd4045ff89b')	200	9.16MiB	39.86KiB	9.06MiB	9.16MiB	9.21MiB	9.23MiB	9.25MiB	9.27MiB	0.00935174
('syslog_log2metric_splunk_hec_metrics', 'baseline', '25a9cf6f-2247-4db8-af15-7fbcbc75ae9b')	200	8.96MiB	54.75KiB	8.83MiB	8.97MiB	9.04MiB	9.05MiB	9.07MiB	9.08MiB	-0.0968558
('syslog_log2metric_splunk_hec_metrics', 'baseline', 'dcfd697f-0397-420d-8441-302cd4b10263')	200	8.91MiB	304.08KiB	8.22MiB	9.06MiB	9.11MiB	9.12MiB	9.13MiB	9.16MiB	-1.34651
('http_to_http_acks', 'comparison', '6ef078dc-e079-4b6e-87a0-daad05e128c0')	199	8.79MiB	3.74MiB	1.85MiB	9.34MiB	13.63MiB	15.42MiB	18.82MiB	19.12MiB	0.382889
('splunk_hec_route_s3', 'baseline', '4433c8b2-45bd-42b0-a761-a620db130f85')	199	8.75MiB	2.01MiB	3.92MiB	8.79MiB	11.46MiB	11.99MiB	12.55MiB	13.27MiB	0.0433449
('splunk_hec_route_s3', 'comparison', '60a2831f-1644-42e2-8e63-14ab91bced28')	199	8.74MiB	2.0MiB	4.41MiB	8.89MiB	11.43MiB	11.76MiB	12.16MiB	12.44MiB	-0.114334
('http_to_http_acks', 'baseline', '3d659fd4-0e73-4046-a97f-751be548fce0')	200	8.74MiB	3.57MiB	1.34MiB	9.19MiB	12.29MiB	15.35MiB	18.93MiB	18.99MiB	0.454034
('splunk_hec_route_s3', 'baseline', 'f73d778b-f746-4ac9-8860-2bf95299bf9b')	199	8.66MiB	1.84MiB	4.98MiB	8.51MiB	11.2MiB	11.97MiB	12.43MiB	12.86MiB	0.187825
('splunk_hec_route_s3', 'comparison', 'eb478d69-46c5-4162-bb95-f88420649605')	199	8.35MiB	1.93MiB	4.87MiB	8.05MiB	11.07MiB	11.71MiB	12.69MiB	13.17MiB	0.362712
('splunk_hec_route_s3', 'comparison', '74dea297-6a49-4680-9cf9-95a37468a43a')	199	8.27MiB	2.0MiB	4.05MiB	8.09MiB	10.97MiB	11.22MiB	12.52MiB	13.29MiB	0.0723339
('splunk_hec_route_s3', 'baseline', 'bd131c48-3e86-47ec-969f-9bc1096f4640')	199	8.27MiB	1.97MiB	4.11MiB	8.25MiB	10.8MiB	11.6MiB	12.14MiB	13.32MiB	0.164008
('syslog_log2metric_humio_metrics', 'baseline', 'f745e0ea-30fa-457b-885c-94f99ac712be')	202	8.12MiB	42.31KiB	8.02MiB	8.12MiB	8.17MiB	8.18MiB	8.21MiB	8.24MiB	-0.0656484
('syslog_log2metric_humio_metrics', 'baseline', '34576788-e8a9-400a-b4bc-4ad6c93ab88e')	202	8.04MiB	31.8KiB	7.96MiB	8.04MiB	8.08MiB	8.09MiB	8.11MiB	8.13MiB	0.141124
('syslog_log2metric_humio_metrics', 'baseline', '33d9a14f-2aac-4b93-8f33-6a3f7cafd555')	202	7.93MiB	34.43KiB	7.85MiB	7.93MiB	7.97MiB	7.99MiB	8.0MiB	8.04MiB	0.304672
('syslog_log2metric_humio_metrics', 'comparison', 'a079cb52-56f3-4360-9337-74f370fdc195')	200	7.86MiB	30.51KiB	7.79MiB	7.87MiB	7.9MiB	7.92MiB	7.93MiB	7.94MiB	0.0261536
('syslog_regex_logs2metric_ddmetrics', 'comparison', 'd49a1a22-4281-4410-9a91-f8170cf1e4a3')	200	7.85MiB	270.04KiB	7.56MiB	7.69MiB	8.29MiB	8.32MiB	8.34MiB	8.39MiB	0.746723
('syslog_regex_logs2metric_ddmetrics', 'comparison', 'f254d704-2575-4630-b0bc-683396fa3c99')	200	7.82MiB	265.38KiB	7.5MiB	7.69MiB	8.25MiB	8.28MiB	8.33MiB	8.34MiB	0.677761
('syslog_log2metric_humio_metrics', 'comparison', '280fae5e-2180-42b3-ab9d-228605e8f32f')	200	7.78MiB	28.09KiB	7.71MiB	7.78MiB	7.82MiB	7.83MiB	7.84MiB	7.85MiB	-0.118769
('syslog_log2metric_humio_metrics', 'comparison', 'e1812ece-f7ca-455b-9cb5-a66a6e30f02e')	200	7.78MiB	37.81KiB	7.69MiB	7.78MiB	7.83MiB	7.84MiB	7.87MiB	7.89MiB	-0.0580663
('syslog_regex_logs2metric_ddmetrics', 'comparison', '3b4a4df0-b8cc-4a58-a4e7-b1e6001c3ce7')	200	7.53MiB	326.01KiB	7.1MiB	7.45MiB	7.98MiB	8.0MiB	8.04MiB	8.05MiB	0.293989
('syslog_regex_logs2metric_ddmetrics', 'baseline', '0d752d15-e803-4ae3-99d6-11e5c2e1e365')	200	7.46MiB	329.18KiB	6.98MiB	7.39MiB	7.88MiB	7.91MiB	7.95MiB	7.97MiB	0.193593
('syslog_regex_logs2metric_ddmetrics', 'baseline', '05c487fa-6604-4173-80b5-4424f8a55b2b')	200	7.28MiB	301.97KiB	6.85MiB	7.29MiB	7.65MiB	7.68MiB	7.72MiB	7.76MiB	-0.00232796
('syslog_loki', 'baseline', '434bdf66-6dfb-4906-bce0-6acf3c2e46dc')	202	7.16MiB	43.94KiB	7.01MiB	7.16MiB	7.21MiB	7.22MiB	7.25MiB	7.26MiB	-0.584598
('syslog_loki', 'baseline', '8845d542-c761-4746-b49c-26909de6f093')	202	6.88MiB	39.07KiB	6.74MiB	6.88MiB	6.92MiB	6.93MiB	6.93MiB	6.97MiB	-0.843664
('syslog_regex_logs2metric_ddmetrics', 'baseline', 'f44e9130-081d-4681-a1e9-7ff60d1d0aa1')	200	6.82MiB	274.92KiB	6.3MiB	6.97MiB	7.06MiB	7.09MiB	7.1MiB	7.13MiB	-0.667835
('syslog_loki', 'baseline', '61ac4794-f85a-4aeb-91b5-9eb7e4331020')	202	6.8MiB	219.03KiB	6.19MiB	6.9MiB	6.98MiB	6.99MiB	7.0MiB	7.01MiB	-1.17876
('syslog_loki', 'comparison', 'e18ecc2a-d36e-4c58-bc63-1a10f0ccf9a4')	200	6.76MiB	42.04KiB	6.62MiB	6.77MiB	6.81MiB	6.82MiB	6.83MiB	6.84MiB	-0.80844
('syslog_loki', 'comparison', '4d1ca9c5-b96c-487a-ab0a-e050065cc08e')	200	6.7MiB	41.95KiB	6.56MiB	6.71MiB	6.74MiB	6.75MiB	6.77MiB	6.78MiB	-1.12536
('syslog_loki', 'comparison', '13648adc-2e21-4761-a648-ca5772c24a2c')	200	6.63MiB	51.45KiB	6.49MiB	6.64MiB	6.7MiB	6.71MiB	6.74MiB	6.75MiB	-0.386377
('splunk_transforms_splunk3', 'comparison', '3a661a13-553d-4ad9-811b-59fe325051df')	202	5.51MiB	1.55MiB	2.92MiB	5.42MiB	7.7MiB	8.12MiB	8.48MiB	8.88MiB	0.293122
('splunk_transforms_splunk3', 'baseline', '2ccad079-663e-440d-8c74-a1a0505cdf1e')	203	5.32MiB	1.48MiB	2.61MiB	5.11MiB	7.33MiB	7.78MiB	8.13MiB	8.63MiB	0.288916
('splunk_transforms_splunk3', 'baseline', '440d9ebe-df35-4bca-91ec-d58355d60d22')	203	5.29MiB	1.46MiB	2.5MiB	5.01MiB	7.36MiB	7.64MiB	8.32MiB	8.7MiB	0.294513
('splunk_transforms_splunk3', 'baseline', '985d6c4a-2c94-4c5c-8a09-d59e054b6f56')	203	5.27MiB	1.48MiB	2.68MiB	5.02MiB	7.28MiB	7.71MiB	8.59MiB	8.76MiB	0.317109
('splunk_transforms_splunk3', 'comparison', 'ca007fab-bacc-469d-ac41-3bf194bc81a6')	202	5.2MiB	1.5MiB	2.23MiB	5.05MiB	7.19MiB	7.51MiB	8.04MiB	8.55MiB	0.129818
('splunk_transforms_splunk3', 'comparison', 'fe852ce9-82da-400a-973e-e29e3b00facc')	202	4.85MiB	1.4MiB	2.34MiB	4.74MiB	6.88MiB	7.4MiB	7.84MiB	7.9MiB	0.335967
('http_datadog_filter_blackhole', 'baseline', 'b512b7e9-593d-4cdb-bff8-9367fafd650d')	201	2.19MiB	877.26KiB	490.01KiB	2.13MiB	3.33MiB	3.65MiB	4.27MiB	4.35MiB	0.279872
('http_datadog_filter_blackhole', 'comparison', 'd43e420a-f7ec-4486-a755-ca56da1c1654')	202	2.19MiB	839.72KiB	506.0KiB	2.05MiB	3.31MiB	3.56MiB	4.22MiB	4.33MiB	0.361677
('http_datadog_filter_blackhole', 'baseline', '88f160e0-e665-4c19-903e-2dc34afbf986')	202	2.19MiB	838.73KiB	384.01KiB	2.11MiB	3.32MiB	3.57MiB	4.09MiB	4.43MiB	0.235142
('http_datadog_filter_blackhole', 'comparison', '55cbc4f9-a4a8-4bc2-8361-416a88e87523')	202	2.18MiB	851.66KiB	260.99KiB	2.11MiB	3.31MiB	3.58MiB	4.12MiB	4.32MiB	0.211973
('http_datadog_filter_blackhole', 'comparison', 'bda198de-15c7-473c-9491-1c498eafbcd6')	202	2.18MiB	835.83KiB	276.98KiB	2.18MiB	3.24MiB	3.61MiB	4.0MiB	4.49MiB	0.236639
('http_datadog_filter_blackhole', 'baseline', 'ef71d034-2c8b-46b3-a03e-810727a96deb')	202	2.14MiB	803.96KiB	276.98KiB	2.07MiB	3.08MiB	3.35MiB	4.06MiB	4.69MiB	0.361864
('http_pipelines_no_grok_blackhole', 'baseline', '627dadd1-bf60-402c-8c8e-378e1fbe8bbd')	200	2.04MiB	826.01KiB	260.99KiB	1.95MiB	3.11MiB	3.63MiB	3.83MiB	4.18MiB	0.338041
('http_pipelines_no_grok_blackhole', 'baseline', 'de914b41-43f7-44aa-8373-889bf73e5d8e')	200	2.01MiB	841.0KiB	260.99KiB	1.84MiB	3.19MiB	3.56MiB	3.83MiB	4.0MiB	0.384609
('http_pipelines_no_grok_blackhole', 'comparison', 'd6b2db8a-b549-4cde-b076-b2690d6e77f8')	200	1.99MiB	850.21KiB	260.99KiB	1.82MiB	3.07MiB	3.54MiB	3.83MiB	4.19MiB	0.338807
('http_pipelines_no_grok_blackhole', 'comparison', '1ed0df38-9c7f-42f5-b260-be4b4d2fd121')	201	1.91MiB	795.6KiB	93.77KiB	1.87MiB	3.19MiB	3.31MiB	3.52MiB	3.67MiB	0.278678
('http_pipelines_no_grok_blackhole', 'comparison', 'e8f98006-9a58-4cb9-ba48-a03305777a8d')	200	1.9MiB	837.53KiB	260.99KiB	1.76MiB	3.06MiB	3.34MiB	3.81MiB	4.19MiB	0.408577
('http_pipelines_no_grok_blackhole', 'baseline', '03e0056d-49bc-4ba0-9bda-60dbcfcfffa4')	201	1.9MiB	825.18KiB	260.99KiB	1.8MiB	2.98MiB	3.29MiB	3.78MiB	4.19MiB	0.331196
('http_pipelines_blackhole_acks', 'comparison', '4661ab0a-41cc-4259-867e-ab1fe3a2a58c')	198	902.23KiB	546.67KiB	31.98KiB	822.11KiB	1.63MiB	1.72MiB	2.38MiB	2.53MiB	0.643915
('http_pipelines_blackhole', 'baseline', 'dd039f6d-ec76-423d-9b78-f080979e6e0b')	196	899.33KiB	544.18KiB	15.99KiB	752.59KiB	1.61MiB	1.71MiB	2.41MiB	2.57MiB	0.751778
('http_pipelines_blackhole_acks', 'comparison', '81836191-8221-4d42-af3c-fabbd3886b57')	199	897.31KiB	531.99KiB	31.98KiB	829.76KiB	1.55MiB	1.74MiB	2.24MiB	2.5MiB	0.576454
('http_pipelines_blackhole_acks', 'baseline', '26ddd13b-24e4-431a-8b00-0dd24715892f')	199	883.7KiB	511.06KiB	62.35KiB	842.56KiB	1.5MiB	1.67MiB	2.38MiB	2.53MiB	0.764581
('http_pipelines_blackhole', 'comparison', 'd483017f-050f-48aa-b6ef-bc1ea3370b3f')	200	836.41KiB	512.6KiB	31.98KiB	721.1KiB	1.57MiB	1.66MiB	2.07MiB	2.43MiB	0.696413
('http_pipelines_blackhole', 'baseline', '588f3576-edd4-4dbb-8448-1c224edae24c')	198	832.77KiB	499.12KiB	15.99KiB	720.84KiB	1.57MiB	1.63MiB	1.99MiB	2.41MiB	0.639393
('http_pipelines_blackhole_acks', 'comparison', 'be6af5e0-711a-4db8-8de1-dbe1b8174edd')	198	827.02KiB	524.13KiB	31.98KiB	737.71KiB	1.59MiB	1.7MiB	2.16MiB	2.5MiB	0.74506
('http_pipelines_blackhole', 'comparison', '86814d0a-2a94-45c8-a25b-e7679c97424d')	199	825.94KiB	533.37KiB	15.99KiB	676.5KiB	1.57MiB	1.67MiB	2.19MiB	2.69MiB	0.869289
('http_pipelines_blackhole', 'baseline', 'dbe0ae88-0fde-4c26-84e0-d9465388068b')	197	816.93KiB	510.76KiB	15.99KiB	675.38KiB	1.56MiB	1.63MiB	2.35MiB	2.98MiB	1.05697
('http_pipelines_blackhole_acks', 'baseline', 'da2a0764-4225-48fe-8677-aa580ce66e71')	200	816.03KiB	506.79KiB	31.98KiB	714.5KiB	1.54MiB	1.69MiB	2.16MiB	2.38MiB	0.752345
('http_pipelines_blackhole_acks', 'baseline', 'bfc0a5ce-a0ff-415a-a89d-eb79d9a03947')	200	805.42KiB	543.68KiB	31.98KiB	706.53KiB	1.54MiB	1.74MiB	2.39MiB	2.5MiB	0.919424
('http_pipelines_blackhole', 'comparison', 'c4723a18-6d08-4225-9777-ab5ea6ec462d')	197	707.36KiB	448.35KiB	31.98KiB	613.02KiB	1.42MiB	1.55MiB	1.69MiB	2.06MiB	0.72902

Jan 14 '22 00:01 github-actions[bot]

Soak Test Results

Baseline: 80c0e0ec8f8a65f850f87c289397e15ee4b801ea Comparison: 2eb664483b5b7c38c490b4f40c74a16c051a3f65 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	comparison mean	comparison stdev	p-value	erratic
fluent_elasticsearch	1.69MiB	2.45	68.93MiB	2.03MiB	70.62MiB	887.69KiB	4.61884e-65	False
datadog_agent_remap_datadog_logs	1.24MiB	3.42	36.2MiB	1.09MiB	37.44MiB	353.72KiB	2.41606e-108	False
splunk_hec_indexer_ack_blackhole	348.53KiB	1.72	19.78MiB	936.37KiB	20.12MiB	961.25KiB	2.87704e-10	False
splunk_hec_route_s3	204.46KiB	2.34	8.52MiB	1.92MiB	8.72MiB	1.96MiB	0.0757453	False
syslog_loki	202.65KiB	2.93	6.75MiB	52.6KiB	6.95MiB	130.23KiB	8.27144e-165	False
syslog_log2metric_splunk_hec_metrics	198.33KiB	2.09	9.28MiB	60.16KiB	9.47MiB	187.09KiB	4.02187e-98	False
splunk_hec_to_splunk_hec_logs_noack	113.14KiB	0.61	18.24MiB	814.39KiB	18.35MiB	912.57KiB	0.0238363	False
syslog_log2metric_humio_metrics	106.73KiB	1.34	7.76MiB	566.51KiB	7.87MiB	335.51KiB	0.000316101	False
syslog_humio_logs	53.49KiB	0.54	9.63MiB	328.41KiB	9.68MiB	365.15KiB	0.0076807	False
splunk_transforms_splunk3	48.66KiB	0.9	5.25MiB	1.49MiB	5.3MiB	1.47MiB	0.575889	False
http_datadog_filter_blackhole	20.04KiB	0.92	2.12MiB	829.66KiB	2.14MiB	816.82KiB	0.671897	False
http_pipelines_blackhole	1.13KiB	0.13	841.95KiB	521.91KiB	843.08KiB	510.61KiB	0.96989	False
http_pipelines_blackhole_acks	-5.35KiB	-0.66	812.4KiB	524.1KiB	807.05KiB	517.83KiB	0.85935	False
http_pipelines_no_grok_blackhole	-41.61KiB	-2.08	1.95MiB	782.54KiB	1.91MiB	826.39KiB	0.36957	False
splunk_hec_to_splunk_hec_logs_acks	-64.63KiB	-0.36	17.52MiB	919.72KiB	17.46MiB	1.04MiB	0.263203	False
syslog_splunk_hec_logs	-123.16KiB	-1.24	9.68MiB	316.54KiB	9.56MiB	500.82KiB	4.30639e-07	False
http_to_http_noack	-256.59KiB	-1.19	20.98MiB	893.41KiB	20.73MiB	915.09KiB	1.06771e-06	False
syslog_regex_logs2metric_ddmetrics	-338.63KiB	-4.32	7.65MiB	453.19KiB	7.32MiB	370.56KiB	3.11997e-35	False
http_to_http_acks	-742.52KiB	-7.31	9.92MiB	4.52MiB	9.19MiB	4.15MiB	0.00389962	False
datadog_agent_remap_datadog_logs_acks	-963.8KiB	-2.57	36.61MiB	882.91KiB	35.67MiB	312.64KiB	4.04701e-102	False
datadog_agent_remap_blackhole	-980.74KiB	-3.47	27.6MiB	770.65KiB	26.64MiB	999.66KiB	9.97763e-71	False
datadog_agent_remap_blackhole_acks	-1.36MiB	-4.8	28.37MiB	713.98KiB	27.01MiB	959.15KiB	4.28898e-135	False
fluent_remap_aws_firehose	-2.02MiB	-2.77	72.78MiB	1.83MiB	70.76MiB	2.41MiB	1.24141e-54	False

Fine details of each soak run.

(experiment, variant, run_id)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_remap_aws_firehose', 'baseline', 'c28fdcba-1789-452a-9e9b-046402ef4a49')	203	74.79MiB	386.54KiB	73.57MiB	74.83MiB	75.24MiB	75.38MiB	75.53MiB	75.68MiB	-0.291845
('fluent_remap_aws_firehose', 'comparison', 'dff104ee-091a-4e18-9e07-02194e10c825')	202	74.04MiB	377.54KiB	73.06MiB	74.06MiB	74.47MiB	74.58MiB	74.83MiB	75.01MiB	-0.222228
('fluent_remap_aws_firehose', 'baseline', '333f668e-01e0-44aa-a5de-5ed063019617')	203	73.11MiB	407.25KiB	71.99MiB	73.15MiB	73.58MiB	73.72MiB	74.03MiB	74.13MiB	-0.0456734
('fluent_elasticsearch', 'comparison', 'c4ba4235-27e6-4d0d-87c0-9661b6ca99b6')	199	71.68MiB	407.29KiB	70.25MiB	71.69MiB	72.14MiB	72.25MiB	72.52MiB	72.66MiB	-0.292592
('fluent_elasticsearch', 'baseline', '431f06c6-5cba-4931-950b-9375c87bf9bc')	199	71.65MiB	423.99KiB	70.71MiB	71.66MiB	72.16MiB	72.3MiB	72.58MiB	72.7MiB	-0.0304655
('fluent_remap_aws_firehose', 'baseline', 'f815384d-ecb4-447a-a339-206de98bdd31')	203	70.44MiB	360.39KiB	69.5MiB	70.47MiB	70.88MiB	70.95MiB	71.15MiB	71.22MiB	-0.248401
('fluent_elasticsearch', 'comparison', '234e65e2-7fc8-4251-92d4-a622f910b73e')	199	70.32MiB	444.6KiB	69.32MiB	70.35MiB	70.83MiB	71.09MiB	71.33MiB	71.83MiB	0.130655
('fluent_elasticsearch', 'comparison', '127f0f4b-82ec-4768-aa39-79b316023a28')	199	69.86MiB	360.22KiB	68.87MiB	69.85MiB	70.38MiB	70.41MiB	70.58MiB	70.87MiB	-0.0433119
('fluent_remap_aws_firehose', 'comparison', 'c4cb801a-d435-4c33-adbe-1091320eb678')	203	69.81MiB	376.48KiB	68.84MiB	69.79MiB	70.26MiB	70.32MiB	70.48MiB	70.76MiB	-0.253823
('fluent_remap_aws_firehose', 'comparison', '268d8e65-542c-4a6f-a486-1ebd5118c520')	203	68.46MiB	513.27KiB	66.65MiB	68.52MiB	69.06MiB	69.16MiB	69.53MiB	69.72MiB	-0.40965
('fluent_elasticsearch', 'baseline', '0cb3bf9b-9d8f-40b0-b210-587161f56a4f')	199	68.04MiB	694.99KiB	66.44MiB	68.19MiB	68.76MiB	68.91MiB	69.12MiB	69.24MiB	-0.664467
('fluent_elasticsearch', 'baseline', 'fbd78995-20b8-48dd-a52e-923383e2e503')	199	67.11MiB	415.61KiB	66.07MiB	67.1MiB	67.6MiB	67.77MiB	68.09MiB	68.18MiB	0.0844973
('datadog_agent_remap_datadog_logs_acks', 'baseline', 'f20d3696-e294-47e5-acec-5d0266fed11b')	199	37.75MiB	247.07KiB	37.17MiB	37.77MiB	38.05MiB	38.1MiB	38.27MiB	38.36MiB	-0.13967
('datadog_agent_remap_datadog_logs', 'comparison', '0665d22f-2145-42c3-9ff1-7ba3d79f13e5')	201	37.59MiB	397.59KiB	33.4MiB	37.6MiB	37.92MiB	37.99MiB	38.28MiB	38.38MiB	-6.25492
('datadog_agent_remap_datadog_logs', 'baseline', '5e590b90-8186-4da1-b140-e6fbd34ec445')	200	37.51MiB	229.54KiB	36.98MiB	37.5MiB	37.8MiB	37.88MiB	37.98MiB	38.15MiB	0.0217997
('datadog_agent_remap_datadog_logs', 'comparison', 'c9921d65-0887-4a64-8a82-75f69fda0867')	200	37.47MiB	297.0KiB	36.7MiB	37.44MiB	37.85MiB	37.91MiB	38.03MiB	38.27MiB	-0.148951
('datadog_agent_remap_datadog_logs', 'comparison', '08a8199d-c891-4a12-a6f4-961c52ce5a88')	200	37.25MiB	261.4KiB	36.52MiB	37.25MiB	37.55MiB	37.68MiB	37.79MiB	38.0MiB	-0.186815
('datadog_agent_remap_datadog_logs_acks', 'baseline', '02952163-e204-4b5c-86d4-1efe2ef7895d')	203	36.16MiB	265.11KiB	35.33MiB	36.19MiB	36.43MiB	36.59MiB	36.68MiB	36.77MiB	-0.526857
('datadog_agent_remap_datadog_logs', 'baseline', '692e3e27-d900-44aa-887e-40b6a117dfcb')	200	35.95MiB	236.69KiB	35.44MiB	35.96MiB	36.25MiB	36.37MiB	36.48MiB	36.53MiB	0.132788
('datadog_agent_remap_datadog_logs_acks', 'baseline', '46076015-daf0-4fb5-9c89-e8f6ddf18edf')	199	35.92MiB	381.83KiB	34.82MiB	35.96MiB	36.37MiB	36.5MiB	36.66MiB	36.7MiB	-0.365764
('datadog_agent_remap_datadog_logs_acks', 'comparison', 'aea27c53-f416-4972-8071-96459695cccc')	202	35.73MiB	394.86KiB	34.6MiB	35.79MiB	36.15MiB	36.24MiB	36.44MiB	36.86MiB	-0.364083
('datadog_agent_remap_datadog_logs_acks', 'comparison', 'e6200cf5-2cc6-4c10-b2f0-4416df9c248a')	202	35.68MiB	244.48KiB	35.1MiB	35.7MiB	35.98MiB	36.08MiB	36.21MiB	36.32MiB	-0.0831199
('datadog_agent_remap_datadog_logs_acks', 'comparison', 'e55b79bd-036f-4997-862f-bd09fee0d9b5')	202	35.59MiB	261.44KiB	34.94MiB	35.57MiB	35.93MiB	35.98MiB	36.15MiB	36.27MiB	-0.00673946
('datadog_agent_remap_datadog_logs', 'baseline', 'ddc1426e-6738-451b-b088-2a3c49ed913f')	201	35.13MiB	767.01KiB	30.97MiB	34.82MiB	36.34MiB	36.46MiB	36.6MiB	36.73MiB	-0.128037
('datadog_agent_remap_blackhole_acks', 'baseline', '4011fd9f-329f-4597-8dba-b5e14bf84e50')	200	29.1MiB	543.0KiB	27.53MiB	29.1MiB	29.78MiB	29.98MiB	30.37MiB	30.68MiB	0.161135
('datadog_agent_remap_blackhole_acks', 'baseline', '66b0b6ed-cc13-4710-81ac-fee0bb2fece6')	200	28.13MiB	376.97KiB	27.2MiB	28.15MiB	28.57MiB	28.66MiB	28.86MiB	29.13MiB	-0.140391
('datadog_agent_remap_blackhole_acks', 'baseline', '3f06339a-f705-428f-9dae-8c69f4658182')	200	27.89MiB	481.61KiB	26.48MiB	27.91MiB	28.43MiB	28.62MiB	28.91MiB	29.13MiB	-0.0997082
('datadog_agent_remap_blackhole_acks', 'comparison', '8404e0f5-d7ed-4bf6-8bcc-7cd72b5ae88c')	200	27.85MiB	663.35KiB	25.86MiB	27.98MiB	28.6MiB	28.74MiB	29.12MiB	29.32MiB	-0.474118
('datadog_agent_remap_blackhole', 'baseline', 'e4ced068-4352-44f4-ae51-88b261b354ef')	203	27.84MiB	567.72KiB	23.99MiB	27.86MiB	28.47MiB	28.61MiB	28.86MiB	28.98MiB	-1.72275
('datadog_agent_remap_blackhole', 'baseline', 'df93a269-b438-40b8-9a6c-e5453ca8cf85')	202	27.66MiB	769.68KiB	25.67MiB	27.61MiB	28.65MiB	28.87MiB	29.15MiB	29.72MiB	0.0387784
('datadog_agent_remap_blackhole', 'comparison', 'e0f8d605-8182-41bc-9860-24bbf1594f93')	201	27.5MiB	657.28KiB	25.59MiB	27.44MiB	28.32MiB	28.5MiB	28.96MiB	29.19MiB	0.0130329
('datadog_agent_remap_blackhole', 'baseline', 'f067178d-34f0-4577-ab3e-26897e9a83fd')	202	27.3MiB	843.74KiB	25.21MiB	27.28MiB	28.34MiB	28.69MiB	29.2MiB	29.54MiB	0.0509849
('datadog_agent_remap_blackhole_acks', 'comparison', '026756fd-0cbf-4b24-b832-5371a9855807')	200	26.97MiB	588.1KiB	23.31MiB	26.96MiB	27.62MiB	27.81MiB	28.19MiB	28.61MiB	-1.19865
('datadog_agent_remap_blackhole', 'comparison', '18f3b7d3-5995-4ea9-8b81-f3d1e91798c8')	201	26.41MiB	966.96KiB	24.14MiB	26.34MiB	27.68MiB	27.93MiB	28.34MiB	28.76MiB	-0.0174075
('datadog_agent_remap_blackhole_acks', 'comparison', 'b886f79b-efeb-4f20-95c1-1ed202f15240')	200	26.21MiB	745.03KiB	24.51MiB	26.12MiB	27.18MiB	27.41MiB	28.06MiB	28.46MiB	0.450258
('datadog_agent_remap_blackhole', 'comparison', 'b9e2d04f-2e73-4f4a-aff9-2956df4e0d40')	201	26.02MiB	632.82KiB	24.28MiB	26.06MiB	26.72MiB	26.93MiB	27.26MiB	28.14MiB	-0.0958031
('http_to_http_noack', 'comparison', '60c4b0cf-bd9d-4144-9dc5-a9e366897a08')	199	21.41MiB	806.6KiB	18.8MiB	21.4MiB	22.35MiB	22.6MiB	23.43MiB	23.72MiB	0.0469937
('http_to_http_noack', 'baseline', 'e2143636-92ce-4d42-9d9b-d9cd924a3612')	200	21.15MiB	835.55KiB	17.13MiB	21.18MiB	22.2MiB	22.34MiB	22.83MiB	23.09MiB	-0.740262
('http_to_http_noack', 'baseline', '61aa1a99-53a3-4a87-a9ec-c2023318ccde')	199	20.95MiB	901.17KiB	18.94MiB	20.91MiB	22.03MiB	22.37MiB	22.98MiB	23.19MiB	0.0688612
('splunk_hec_indexer_ack_blackhole', 'comparison', 'cf5dfd81-c9af-4f71-b7d8-df504f6a67a4')	200	20.84MiB	760.02KiB	19.23MiB	20.75MiB	21.79MiB	22.07MiB	22.42MiB	24.12MiB	0.564983
('http_to_http_noack', 'baseline', '3c89fd46-2d86-4295-a0f4-8ba939aca2f3')	199	20.83MiB	914.97KiB	17.51MiB	20.81MiB	21.94MiB	22.21MiB	22.73MiB	23.35MiB	-0.0454353
('http_to_http_noack', 'comparison', '2303723e-0826-4f07-bc88-ee801557ca24')	199	20.49MiB	756.9KiB	18.65MiB	20.52MiB	21.38MiB	21.64MiB	22.3MiB	22.7MiB	0.0675318
('splunk_hec_indexer_ack_blackhole', 'baseline', '1aa40df4-cf86-4a93-99f5-6407d441dc8d')	200	20.47MiB	776.57KiB	18.64MiB	20.42MiB	21.44MiB	21.71MiB	22.5MiB	22.63MiB	0.184352
('http_to_http_noack', 'comparison', 'c80f62f5-7385-4d2b-9c9b-5e818202db9d')	199	20.29MiB	734.04KiB	17.99MiB	20.36MiB	21.15MiB	21.37MiB	21.66MiB	21.92MiB	-0.402539
('splunk_hec_indexer_ack_blackhole', 'comparison', '17f96994-2d71-4be3-b941-a078df6d2bff')	200	19.84MiB	839.11KiB	17.89MiB	19.8MiB	20.92MiB	21.29MiB	21.71MiB	21.92MiB	0.140882
('splunk_hec_indexer_ack_blackhole', 'comparison', '233919be-322e-454d-aef4-a8faa6253d5b')	200	19.68MiB	819.42KiB	17.43MiB	19.76MiB	20.7MiB	21.04MiB	21.5MiB	21.9MiB	-0.126544
('splunk_hec_indexer_ack_blackhole', 'baseline', '0c5149da-7fe0-47f7-bd53-35125f78b96e')	199	19.51MiB	826.52KiB	16.3MiB	19.62MiB	20.36MiB	20.59MiB	21.28MiB	21.59MiB	-0.57499
('splunk_hec_indexer_ack_blackhole', 'baseline', '2d3ad5e0-c11a-4fa4-81c6-bd8ebb2c2585')	200	19.35MiB	758.02KiB	17.18MiB	19.38MiB	20.26MiB	20.43MiB	21.26MiB	21.41MiB	-0.149094
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', 'c408c500-7163-4405-ae96-56f4586990f2')	199	19.11MiB	654.37KiB	17.43MiB	19.07MiB	19.94MiB	20.11MiB	20.37MiB	20.6MiB	-0.178225
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', 'e661b1d3-3089-44fe-8e1f-1ddaa4b69eb0')	200	18.61MiB	738.29KiB	16.78MiB	18.62MiB	19.5MiB	19.67MiB	20.12MiB	20.4MiB	-0.164044
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', '39e90bcb-8add-45d6-8d02-d538ff47efec')	200	18.43MiB	689.06KiB	16.81MiB	18.47MiB	19.32MiB	19.49MiB	19.7MiB	20.01MiB	-0.160098
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', 'd5ab7a6d-7994-43ab-8f66-47b451d14d77')	198	18.12MiB	898.02KiB	15.61MiB	18.28MiB	19.17MiB	19.31MiB	19.88MiB	20.13MiB	-0.548055
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', '0ba53d03-c2ba-4ad9-8ba8-127a16b73e9d')	199	18.09MiB	734.67KiB	16.12MiB	18.13MiB	18.94MiB	19.14MiB	19.79MiB	19.93MiB	-0.239528
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', '8c1bbc9a-5c01-42e7-a843-9f1604847831')	200	17.94MiB	831.1KiB	15.66MiB	17.98MiB	18.94MiB	19.16MiB	19.34MiB	19.46MiB	-0.445652
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', '4d01c7fc-2c8c-407a-8b35-93a80d484281')	199	17.86MiB	780.64KiB	15.73MiB	17.83MiB	18.82MiB	19.12MiB	19.53MiB	19.62MiB	-0.128987
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', 'fcf5a0ae-9c36-49ed-a40e-91bc30a15984')	200	17.69MiB	694.81KiB	15.94MiB	17.73MiB	18.51MiB	18.77MiB	19.31MiB	19.55MiB	-0.0707189
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', 'ac579f65-2497-4620-a936-3f419fd6b7e9')	198	17.42MiB	922.9KiB	15.32MiB	17.36MiB	18.73MiB	18.91MiB	19.12MiB	19.43MiB	0.0984246
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', 'da331dda-743b-4e14-8a45-cf98e89ecf7d')	200	17.32MiB	944.27KiB	14.76MiB	17.36MiB	18.53MiB	18.77MiB	18.91MiB	18.98MiB	-0.313126
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', '41aa4e0c-147d-47c7-a39a-04be775085c4')	200	17.31MiB	832.89KiB	13.87MiB	17.3MiB	18.39MiB	18.59MiB	19.13MiB	19.31MiB	-0.327531
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', '21600236-d9e5-4a83-a61b-54c900074e94')	198	16.84MiB	953.13KiB	14.18MiB	16.83MiB	18.16MiB	18.34MiB	18.56MiB	18.73MiB	-0.125953
('http_to_http_acks', 'baseline', 'f572a326-1e11-45a7-b46a-a95c30ac42fb')	200	10.31MiB	4.58MiB	1.58MiB	10.38MiB	17.55MiB	18.71MiB	20.05MiB	20.05MiB	0.22611
('syslog_splunk_hec_logs', 'baseline', 'ae02f3a7-6053-4239-ad80-cce7ab128049')	199	10.01MiB	45.09KiB	9.88MiB	10.01MiB	10.06MiB	10.07MiB	10.1MiB	10.11MiB	-0.522913
('syslog_humio_logs', 'comparison', 'dd968220-c317-4505-84dc-67e4f73f3155')	199	9.97MiB	43.12KiB	9.88MiB	9.96MiB	10.02MiB	10.03MiB	10.05MiB	10.07MiB	-0.0584105
('syslog_splunk_hec_logs', 'comparison', '5f545f40-3893-46c4-bd62-d79702ac2910')	200	9.92MiB	29.2KiB	9.85MiB	9.91MiB	9.95MiB	9.96MiB	9.98MiB	9.99MiB	0.139815
('syslog_humio_logs', 'comparison', '68974b89-17de-4cd4-a1aa-ecb64c8a47b1')	199	9.9MiB	41.86KiB	9.79MiB	9.9MiB	9.95MiB	9.96MiB	9.99MiB	10.0MiB	-0.10681
('syslog_splunk_hec_logs', 'comparison', '202003e4-89f9-4dd1-b605-9407a70538ff')	200	9.89MiB	27.35KiB	9.83MiB	9.89MiB	9.93MiB	9.93MiB	9.95MiB	9.96MiB	0.00670381
('syslog_humio_logs', 'baseline', 'b7f8f3d3-5f37-4780-83ef-66d0b25fdafd')	202	9.88MiB	45.72KiB	9.76MiB	9.88MiB	9.93MiB	9.96MiB	9.98MiB	10.01MiB	0.102592
('syslog_humio_logs', 'baseline', 'b4de17f7-95c1-4f9c-a410-5ac982f060bd')	202	9.83MiB	59.5KiB	9.72MiB	9.83MiB	9.91MiB	9.94MiB	9.97MiB	9.99MiB	0.438427
('syslog_splunk_hec_logs', 'baseline', '53d02f30-6c9e-4bdf-879b-cbcdac2cde76')	199	9.76MiB	32.62KiB	9.65MiB	9.76MiB	9.8MiB	9.81MiB	9.84MiB	9.86MiB	-0.011775
('http_to_http_acks', 'baseline', '684ef026-3e81-4406-8b16-fe6066f5b7aa')	200	9.76MiB	4.58MiB	1.85MiB	9.63MiB	16.35MiB	18.99MiB	20.62MiB	21.36MiB	0.527795
('syslog_log2metric_splunk_hec_metrics', 'comparison', '5e2b8abd-e956-4845-873a-77b5a0ae9c3e')	200	9.7MiB	32.62KiB	9.59MiB	9.71MiB	9.74MiB	9.75MiB	9.77MiB	9.78MiB	-0.555518
('http_to_http_acks', 'baseline', '530877d9-733d-45fa-8f93-d6494f0ac367')	200	9.69MiB	4.39MiB	1.34MiB	9.54MiB	15.6MiB	17.97MiB	19.65MiB	20.32MiB	0.384812
('syslog_log2metric_splunk_hec_metrics', 'comparison', 'f621fc95-dbf1-45bd-82fb-19deb61a1d74')	200	9.45MiB	36.21KiB	9.35MiB	9.45MiB	9.49MiB	9.5MiB	9.53MiB	9.54MiB	0.0391487
('syslog_log2metric_splunk_hec_metrics', 'baseline', '128219df-e781-4f1f-9cab-54431454db78')	200	9.35MiB	29.85KiB	9.27MiB	9.34MiB	9.39MiB	9.4MiB	9.4MiB	9.41MiB	0.0305695
('syslog_splunk_hec_logs', 'baseline', '70957c4e-bc1b-4cfd-a5c0-593878ec08c8')	199	9.28MiB	68.64KiB	9.11MiB	9.27MiB	9.37MiB	9.39MiB	9.41MiB	9.44MiB	0.057453
('syslog_log2metric_splunk_hec_metrics', 'comparison', '99ae53a0-3c03-4c87-9800-789b0faa8067')	200	9.27MiB	35.83KiB	9.09MiB	9.27MiB	9.3MiB	9.31MiB	9.33MiB	9.34MiB	-1.49473
('http_to_http_acks', 'comparison', '1806ac5d-6ccc-4387-8aa3-5f16b06d0798')	199	9.26MiB	3.94MiB	2.41MiB	9.34MiB	14.98MiB	17.45MiB	18.96MiB	19.29MiB	0.534149
('syslog_log2metric_splunk_hec_metrics', 'baseline', '81bbab4d-0717-45c3-aa44-04a8892d7077')	183	9.24MiB	33.63KiB	9.12MiB	9.24MiB	9.28MiB	9.29MiB	9.31MiB	9.31MiB	-0.694209
('syslog_log2metric_splunk_hec_metrics', 'baseline', '780f652b-5cd7-4955-986d-a5334d40a15d')	200	9.24MiB	26.49KiB	9.14MiB	9.24MiB	9.27MiB	9.28MiB	9.29MiB	9.3MiB	-0.10449
('syslog_humio_logs', 'comparison', '79fcc394-5338-4a28-b2af-e81cd9aa84f6')	199	9.19MiB	62.61KiB	8.97MiB	9.19MiB	9.27MiB	9.28MiB	9.31MiB	9.33MiB	-0.418375
('syslog_humio_logs', 'baseline', 'b6e8d31c-c97c-4068-8514-830c258384cb')	202	9.19MiB	59.64KiB	9.03MiB	9.19MiB	9.26MiB	9.27MiB	9.29MiB	9.3MiB	-0.331542
('http_to_http_acks', 'comparison', '9953a1cc-7676-4819-9fff-65528982ca44')	199	9.17MiB	4.12MiB	1.73MiB	9.26MiB	15.29MiB	17.65MiB	18.93MiB	20.12MiB	0.484249
('http_to_http_acks', 'comparison', 'dd935586-d593-4ae3-aa9d-a5f6f56448b9')	199	9.15MiB	4.4MiB	1.62MiB	9.11MiB	15.58MiB	17.97MiB	19.39MiB	21.37MiB	0.593634
('syslog_splunk_hec_logs', 'comparison', '44e3b57f-5328-4f11-8563-cc5d823952d3')	200	8.88MiB	154.52KiB	8.49MiB	8.88MiB	9.07MiB	9.15MiB	9.22MiB	9.3MiB	0.0528315
('splunk_hec_route_s3', 'comparison', '8700e434-6fcf-4627-ac48-d8d407e632c7')	199	8.77MiB	1.92MiB	5.04MiB	8.7MiB	11.48MiB	11.96MiB	12.5MiB	12.83MiB	0.156968
('splunk_hec_route_s3', 'comparison', 'e01e08ce-3b6b-4216-8d44-c0fdeefc039a')	199	8.77MiB	1.96MiB	5.26MiB	8.48MiB	11.49MiB	11.95MiB	12.7MiB	12.93MiB	0.214312
('splunk_hec_route_s3', 'baseline', '17deb1a0-4d1d-409c-b9be-131068bb52ba')	199	8.71MiB	2.01MiB	4.75MiB	8.63MiB	11.65MiB	12.02MiB	12.69MiB	13.13MiB	0.172301
('splunk_hec_route_s3', 'baseline', '6c914bad-8f73-418f-a75c-e4d4b6c1d84d')	199	8.7MiB	1.82MiB	5.12MiB	8.69MiB	11.33MiB	11.65MiB	12.32MiB	13.01MiB	0.133319
('splunk_hec_route_s3', 'comparison', '0cb6e988-c33c-4a93-b756-6d1022bb0ac8')	199	8.62MiB	2.0MiB	4.98MiB	8.45MiB	11.41MiB	11.96MiB	12.54MiB	13.42MiB	0.265385
('syslog_log2metric_humio_metrics', 'comparison', '7e328717-02cc-4e21-96cb-1e552779a565')	200	8.28MiB	33.54KiB	8.19MiB	8.28MiB	8.32MiB	8.33MiB	8.35MiB	8.36MiB	-0.28345
('splunk_hec_route_s3', 'baseline', 'b083a4ef-9b01-4d88-ab6e-6ee4828e8aaa')	199	8.14MiB	1.88MiB	4.5MiB	8.0MiB	10.78MiB	11.21MiB	11.7MiB	12.69MiB	0.137814
('syslog_regex_logs2metric_ddmetrics', 'baseline', 'd9144a6e-eeb9-42e6-961b-636bdaaf3a12')	183	7.79MiB	233.49KiB	7.44MiB	7.8MiB	8.13MiB	8.17MiB	8.3MiB	8.35MiB	0.383619
('syslog_log2metric_humio_metrics', 'baseline', '92c3493f-456c-4d3e-b11c-21925f12fbb0')	202	7.79MiB	34.65KiB	7.7MiB	7.79MiB	7.83MiB	7.84MiB	7.88MiB	7.9MiB	-0.121537
('syslog_log2metric_humio_metrics', 'baseline', 'a7e1eb6f-8b09-471e-9080-6627ff458f3e')	202	7.75MiB	41.5KiB	7.63MiB	7.76MiB	7.81MiB	7.81MiB	7.83MiB	7.83MiB	-0.153906
('syslog_log2metric_humio_metrics', 'baseline', 'e3b1ae2a-c2da-4402-b4c0-86c5754297a1')	71	7.71MiB	1.43MiB	12.83KiB	8.0MiB	8.07MiB	8.09MiB	8.11MiB	8.12MiB	-4.95082
('syslog_regex_logs2metric_ddmetrics', 'baseline', '36f209b7-02e7-4d8d-b3a9-bf02ccd27308')	200	7.71MiB	306.53KiB	7.31MiB	7.56MiB	8.16MiB	8.19MiB	8.22MiB	8.27MiB	0.468077
('syslog_log2metric_humio_metrics', 'comparison', 'cc2102d7-836a-4cf4-8745-81b56a28f90b')	180	7.66MiB	172.22KiB	7.33MiB	7.66MiB	7.89MiB	7.93MiB	7.99MiB	7.99MiB	0.016178
('syslog_log2metric_humio_metrics', 'comparison', '4c2320e3-77b5-4a8d-81a3-25f437ae6358')	200	7.64MiB	157.96KiB	7.24MiB	7.65MiB	7.84MiB	7.89MiB	7.93MiB	7.95MiB	-0.162802
('syslog_regex_logs2metric_ddmetrics', 'comparison', '9c345cb5-79bb-4d11-b9bb-f91951dbf117')	200	7.58MiB	312.89KiB	7.03MiB	7.47MiB	8.03MiB	8.05MiB	8.09MiB	8.12MiB	0.351529
('syslog_regex_logs2metric_ddmetrics', 'comparison', 'da52c45d-48dc-4ce9-bcef-71197c8b1c2a')	200	7.27MiB	314.14KiB	6.82MiB	7.27MiB	7.64MiB	7.67MiB	7.69MiB	7.72MiB	-0.0260546
('syslog_regex_logs2metric_ddmetrics', 'comparison', 'b65e3878-f2c0-49d4-b7eb-9dd450882eea')	200	7.11MiB	309.97KiB	6.52MiB	7.2MiB	7.45MiB	7.5MiB	7.53MiB	7.58MiB	-0.218544
('syslog_regex_logs2metric_ddmetrics', 'baseline', 'fc75b53b-8ae0-4598-b1e5-ddac4bf8620e')	70	7.09MiB	728.55KiB	2.07MiB	7.15MiB	7.58MiB	7.61MiB	7.67MiB	7.67MiB	-5.2654
('syslog_loki', 'comparison', '8dbc1b72-2f56-4c12-8a8b-6e6f6d291985')	200	7.06MiB	73.65KiB	6.84MiB	7.07MiB	7.15MiB	7.17MiB	7.18MiB	7.19MiB	-0.573293
('syslog_loki', 'comparison', '9d44ab11-23bc-49fd-878b-f70aceca5bb2')	200	7.0MiB	38.3KiB	6.84MiB	7.0MiB	7.04MiB	7.05MiB	7.06MiB	7.08MiB	-0.95617
('syslog_loki', 'comparison', 'd8d809d8-f0df-4565-ad30-a9b248900755')	200	6.79MiB	50.02KiB	6.63MiB	6.81MiB	6.85MiB	6.86MiB	6.88MiB	6.9MiB	-0.833739
('syslog_loki', 'baseline', 'c02fe579-a7c4-4e92-8d24-9ec63f259d8d')	202	6.77MiB	48.57KiB	6.64MiB	6.78MiB	6.82MiB	6.83MiB	6.85MiB	6.86MiB	-0.553428
('syslog_loki', 'baseline', '98ee0937-65a4-4321-a398-d725baa506c7')	202	6.76MiB	48.96KiB	6.63MiB	6.77MiB	6.82MiB	6.83MiB	6.85MiB	6.87MiB	-0.627399
('syslog_loki', 'baseline', '51575f40-1ab8-4ed4-959e-de03584b625a')	202	6.73MiB	50.78KiB	6.6MiB	6.74MiB	6.78MiB	6.79MiB	6.81MiB	6.82MiB	-0.673294
('splunk_transforms_splunk3', 'comparison', 'fba40b16-8ec2-4b28-8457-d3cf483cb331')	202	5.32MiB	1.46MiB	2.52MiB	5.13MiB	7.29MiB	7.73MiB	8.42MiB	8.73MiB	0.235867
('splunk_transforms_splunk3', 'comparison', '7d25c015-2f88-48db-84d7-f70ce1e715e3')	202	5.32MiB	1.48MiB	2.64MiB	5.02MiB	7.36MiB	7.7MiB	8.49MiB	8.83MiB	0.328586
('splunk_transforms_splunk3', 'baseline', '2f6a0682-dc93-43e8-bd87-b97d94e73e21')	203	5.26MiB	1.49MiB	2.73MiB	5.04MiB	7.24MiB	7.68MiB	8.31MiB	8.76MiB	0.347939
('splunk_transforms_splunk3', 'comparison', '9bad495d-4d5f-46b0-a1b7-609565042cd4')	203	5.26MiB	1.48MiB	2.29MiB	5.07MiB	7.23MiB	7.64MiB	8.18MiB	8.7MiB	0.181781
('splunk_transforms_splunk3', 'baseline', '92ea85ea-6012-4e1a-abc1-a0ed68cd3a18')	203	5.26MiB	1.49MiB	2.86MiB	4.89MiB	7.39MiB	7.65MiB	8.64MiB	9.18MiB	0.432297
('splunk_transforms_splunk3', 'baseline', '22b4e0fb-a281-41b6-9eca-e743a4c56d4f')	203	5.23MiB	1.5MiB	2.79MiB	4.98MiB	7.31MiB	7.51MiB	8.52MiB	8.64MiB	0.287314
('http_datadog_filter_blackhole', 'baseline', '89723fc8-b758-4f04-9bc3-d741dec77167')	202	2.17MiB	815.99KiB	490.01KiB	2.11MiB	3.27MiB	3.57MiB	4.1MiB	4.39MiB	0.335287
('http_datadog_filter_blackhole', 'comparison', '4691b6a5-8df9-4ce4-a532-f1a2e0e07603')	201	2.15MiB	851.28KiB	260.99KiB	2.05MiB	3.32MiB	3.57MiB	4.26MiB	4.43MiB	0.348552
('http_datadog_filter_blackhole', 'comparison', '422a69bb-9fd3-41e8-b2a7-389247b4af26')	202	2.14MiB	828.86KiB	477.78KiB	2.06MiB	3.24MiB	3.67MiB	3.99MiB	4.49MiB	0.369825
('http_datadog_filter_blackhole', 'comparison', 'cfb6acae-b24f-4dca-9502-d63bcb793cc4')	202	2.13MiB	772.47KiB	276.98KiB	2.04MiB	3.15MiB	3.34MiB	3.91MiB	3.99MiB	0.187292
('http_datadog_filter_blackhole', 'baseline', '2d225b4b-3111-4543-8056-b29a43988e8a')	202	2.13MiB	819.92KiB	260.99KiB	2.06MiB	3.22MiB	3.57MiB	3.95MiB	4.13MiB	0.269949
('http_datadog_filter_blackhole', 'baseline', '4dbb21d3-670b-4069-a46d-3b18cfd67b50')	202	2.06MiB	853.21KiB	384.01KiB	1.98MiB	3.21MiB	3.51MiB	3.9MiB	4.43MiB	0.295311
('http_pipelines_no_grok_blackhole', 'baseline', '2a8de432-4104-4034-86f9-05933e8078af')	200	1.99MiB	737.62KiB	490.01KiB	1.91MiB	3.11MiB	3.17MiB	3.53MiB	3.99MiB	0.212427
('http_pipelines_no_grok_blackhole', 'baseline', 'b948d6ed-9e2c-444b-8848-6f434912dce2')	200	1.99MiB	819.23KiB	276.98KiB	1.88MiB	3.1MiB	3.51MiB	3.82MiB	4.19MiB	0.34286
('http_pipelines_no_grok_blackhole', 'comparison', 'eb762778-2277-4e63-8151-c6c3e641149c')	200	1.94MiB	840.85KiB	245.0KiB	1.85MiB	3.07MiB	3.36MiB	3.83MiB	5.16MiB	0.553471
('http_pipelines_no_grok_blackhole', 'comparison', '24240db8-3b9e-4158-a5f9-32014f92e86c')	205	1.94MiB	816.66KiB	245.0KiB	1.83MiB	3.05MiB	3.31MiB	3.78MiB	3.81MiB	0.293121
('http_pipelines_no_grok_blackhole', 'baseline', 'bd2fd8de-a051-4bb8-92a7-2d1ecd269c4e')	200	1.89MiB	788.04KiB	276.98KiB	1.75MiB	3.03MiB	3.29MiB	3.72MiB	3.81MiB	0.352633
('http_pipelines_no_grok_blackhole', 'comparison', '6d59874c-093f-4758-9dbc-00a3a23d3f50')	200	1.85MiB	822.61KiB	124.71KiB	1.74MiB	2.92MiB	3.35MiB	3.72MiB	3.81MiB	0.36111
('http_pipelines_blackhole', 'comparison', '709172db-5abf-4455-a4e8-1a13edba17ca')	200	877.91KiB	512.71KiB	31.98KiB	796.76KiB	1.57MiB	1.67MiB	2.08MiB	2.54MiB	0.634691
('http_pipelines_blackhole_acks', 'comparison', '7e446b17-9886-4dc9-a0f8-2a7e6f97583e')	200	877.59KiB	562.87KiB	31.98KiB	776.04KiB	1.56MiB	1.74MiB	2.5MiB	3.05MiB	0.924304
('http_pipelines_blackhole', 'baseline', 'ab4f8b2e-067b-4ad7-880a-2067649a7e31')	198	876.07KiB	508.18KiB	31.98KiB	750.39KiB	1.6MiB	1.64MiB	2.09MiB	2.41MiB	0.623862
('http_pipelines_blackhole_acks', 'baseline', '28d724ba-bdbf-427c-b700-9f2272c91b44')	199	846.44KiB	515.86KiB	31.98KiB	766.94KiB	1.58MiB	1.67MiB	2.02MiB	2.53MiB	0.573
('http_pipelines_blackhole', 'comparison', '39b9afa0-d070-4633-bced-60f36192e77c')	201	839.85KiB	492.5KiB	15.99KiB	736.57KiB	1.57MiB	1.64MiB	2.06MiB	2.07MiB	0.523687
('http_pipelines_blackhole', 'baseline', 'd8a73152-3489-4bf0-8fbb-979345f49daf')	197	832.64KiB	509.11KiB	15.99KiB	720.58KiB	1.57MiB	1.66MiB	2.07MiB	2.58MiB	0.76981
('http_pipelines_blackhole', 'baseline', '5cbd1e44-167e-4d96-aee4-4fe9b7fad3f7')	195	816.71KiB	548.61KiB	15.99KiB	659.31KiB	1.58MiB	1.69MiB	2.21MiB	2.88MiB	1.09968
('http_pipelines_blackhole', 'comparison', '35d2d5b1-daa9-42f5-862e-c53eddb2bc7c')	199	811.35KiB	526.6KiB	15.99KiB	678.62KiB	1.53MiB	1.66MiB	2.41MiB	3.04MiB	1.08112
('http_pipelines_blackhole_acks', 'baseline', 'a9a4f9fe-c7b5-40fe-b51d-76f45057815a')	198	804.69KiB	563.67KiB	31.98KiB	675.9KiB	1.54MiB	1.67MiB	2.15MiB	3.24MiB	1.06025
('http_pipelines_blackhole_acks', 'baseline', 'b87b31b0-2b8e-42d4-8786-ff075e9ef2ca')	198	785.89KiB	491.0KiB	31.98KiB	675.64KiB	1.48MiB	1.61MiB	1.97MiB	2.5MiB	0.662199
('http_pipelines_blackhole_acks', 'comparison', 'fbd02dee-57a8-4973-b0ee-b11db5d19032')	199	772.69KiB	487.01KiB	31.98KiB	706.27KiB	1.42MiB	1.58MiB	1.74MiB	2.5MiB	0.570577
('http_pipelines_blackhole_acks', 'comparison', 'c14ce8b6-5f38-4e7a-ae46-0b1ee63a44f0')	199	770.52KiB	495.31KiB	31.98KiB	675.9KiB	1.53MiB	1.59MiB	1.73MiB	2.05MiB	0.45951

Jan 14 '22 18:01 github-actions[bot]

Soak Test Results

Baseline: 48757c17f11cd162cbef4abbc8ea7505bb9c1d2e Comparison: 5c2315584c04c92e449bdfabb03678df51e60946 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

experiment	Δ mean	Δ mean %	confidence
http_to_http_acks	969.29KiB	10.15	99.98%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	comparison mean	comparison stdev	p-value	erratic
http_to_http_acks	969.29KiB	10.15	9.33MiB	4.07MiB	10.28MiB	4.71MiB	0.0002095	False
syslog_log2metric_humio_metrics	475.32KiB	6.17	7.52MiB	768.11KiB	7.99MiB	639.08KiB	5.73796e-24	False
http_pipelines_blackhole_acks	35.69KiB	4.39	813.5KiB	503.35KiB	849.19KiB	524.88KiB	0.230484	False
datadog_agent_remap_datadog_logs	1.59MiB	4.39	36.27MiB	1.11MiB	37.86MiB	414.46KiB	2.69588e-149	False
syslog_regex_logs2metric_ddmetrics	319.3KiB	4.26	7.32MiB	322.3KiB	7.63MiB	323.09KiB	4.67676e-59	False
syslog_humio_logs	395.54KiB	3.98	9.72MiB	262.99KiB	10.1MiB	729.97KiB	2.04013e-26	False
http_pipelines_no_grok_blackhole	73.26KiB	3.76	1.9MiB	799.86KiB	1.97MiB	811.95KiB	0.115635	False
datadog_agent_remap_blackhole	910.27KiB	3.39	26.22MiB	1.3MiB	27.11MiB	800.96KiB	8.83222e-43	False
syslog_log2metric_splunk_hec_metrics	302.98KiB	3.19	9.28MiB	42.67KiB	9.58MiB	159.34KiB	2.75555e-206	False
splunk_hec_indexer_ack_blackhole	440.85KiB	2.23	19.34MiB	851.49KiB	19.77MiB	771.09KiB	2.70525e-20	False
splunk_hec_to_splunk_hec_logs_noack	128.14KiB	0.72	17.49MiB	823.08KiB	17.62MiB	930.29KiB	0.0117522	False
splunk_hec_to_splunk_hec_logs_acks	93.42KiB	0.53	17.19MiB	991.34KiB	17.29MiB	896.08KiB	0.0878182	False
fluent_elasticsearch	244.72KiB	0.34	69.7MiB	2.22MiB	69.94MiB	1.55MiB	0.0310168	False
splunk_transforms_splunk3	6.1KiB	0.11	5.33MiB	1.52MiB	5.33MiB	1.47MiB	0.944617	False
splunk_hec_route_s3	-38.1KiB	-0.43	8.71MiB	2.02MiB	8.68MiB	1.93MiB	0.74503	False
http_pipelines_blackhole	-4.92KiB	-0.61	809.95KiB	489.94KiB	805.03KiB	491.56KiB	0.862723	False
fluent_remap_aws_firehose	-534.51KiB	-0.73	71.96MiB	1.03MiB	71.44MiB	2.04MiB	2.54675e-08	False
syslog_splunk_hec_logs	-108.15KiB	-1.05	10.1MiB	91.11KiB	9.99MiB	710.9KiB	0.000973045	False
datadog_agent_remap_datadog_logs_acks	-1.03MiB	-2.83	36.58MiB	931.25KiB	35.55MiB	413.28KiB	2.24908e-106	False
http_to_http_noack	-715.9KiB	-3.22	21.71MiB	863.3KiB	21.01MiB	986.85KiB	6.71062e-38	False
syslog_loki	-227.68KiB	-3.24	6.86MiB	167.96KiB	6.63MiB	257.82KiB	4.14114e-64	False
http_datadog_filter_blackhole	-98.56KiB	-4.28	2.25MiB	826.79KiB	2.15MiB	811.55KiB	0.0363588	False
datadog_agent_remap_blackhole_acks	-1.61MiB	-5.65	28.49MiB	561.86KiB	26.88MiB	1.23MiB	2.85647e-130	False

Fine details of each soak run.

(experiment, variant, run_id)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_remap_aws_firehose', 'comparison', 'c000c357-a274-4c65-9005-40a98c2a602b')	202	74.27MiB	442.94KiB	73.16MiB	74.3MiB	74.86MiB	74.95MiB	75.13MiB	75.2MiB	-0.23918
('fluent_remap_aws_firehose', 'baseline', '3f7b0e6d-3a04-4d2b-b873-8703d0230b5b')	203	72.79MiB	362.54KiB	71.77MiB	72.8MiB	73.21MiB	73.33MiB	73.67MiB	73.76MiB	-0.0943316
('fluent_remap_aws_firehose', 'baseline', '661f513e-b118-4e35-a4cf-01063872903a')	201	72.46MiB	587.67KiB	70.07MiB	72.5MiB	73.12MiB	73.22MiB	73.41MiB	73.52MiB	-0.872506
('fluent_elasticsearch', 'baseline', 'b571d0df-1922-40ed-821b-95ae43bd903a')	199	71.5MiB	424.92KiB	70.41MiB	71.49MiB	72.05MiB	72.19MiB	72.34MiB	72.41MiB	-0.0905193
('fluent_elasticsearch', 'comparison', 'dea688a4-e620-4de5-af5e-04153b4b62e2')	199	71.47MiB	379.55KiB	70.54MiB	71.47MiB	71.94MiB	72.08MiB	72.19MiB	72.34MiB	-0.108311
('fluent_elasticsearch', 'baseline', '5747b6fb-4d82-406b-9d69-30639668bae0')	199	70.69MiB	1.55MiB	66.04MiB	71.27MiB	71.91MiB	72.05MiB	72.18MiB	72.27MiB	-1.49402
('fluent_remap_aws_firehose', 'baseline', 'a9ad260a-2760-41a1-8f02-1449f50a4aed')	202	70.65MiB	297.73KiB	69.84MiB	70.68MiB	71.0MiB	71.05MiB	71.25MiB	71.28MiB	-0.226189
('fluent_elasticsearch', 'comparison', '288bce64-1d40-4286-8c8f-9846ae8cdfdc')	199	70.44MiB	430.43KiB	69.18MiB	70.46MiB	70.96MiB	71.14MiB	71.3MiB	71.72MiB	-0.0512447
('fluent_remap_aws_firehose', 'comparison', '02be4ecf-39bf-4973-a2b0-7fb29085a77e')	203	70.24MiB	330.3KiB	69.2MiB	70.27MiB	70.62MiB	70.75MiB	70.9MiB	71.0MiB	-0.291844
('fluent_remap_aws_firehose', 'comparison', '95d3e306-1548-432b-b7cc-1eb1e19c8d4d')	203	69.83MiB	485.21KiB	68.29MiB	69.83MiB	70.36MiB	70.55MiB	70.89MiB	71.03MiB	-0.399874
('fluent_elasticsearch', 'comparison', '1c4a4791-4fc4-4292-aea3-f09f79d6d6a2')	199	67.9MiB	329.28KiB	66.78MiB	67.91MiB	68.3MiB	68.42MiB	68.68MiB	68.69MiB	-0.103253
('fluent_elasticsearch', 'baseline', 'e691be7a-e7e8-4292-933e-16cc8435e9aa')	199	66.91MiB	422.54KiB	65.73MiB	66.9MiB	67.48MiB	67.57MiB	67.72MiB	67.74MiB	-0.104046
('datadog_agent_remap_datadog_logs', 'comparison', '300c3850-2770-4fc5-a6d2-528da017552f')	200	38.05MiB	228.72KiB	37.44MiB	38.06MiB	38.31MiB	38.4MiB	38.56MiB	38.7MiB	0.00543574
('datadog_agent_remap_datadog_logs', 'comparison', '630d19c5-d218-4d31-b2a1-06b842ba3970')	201	37.96MiB	459.42KiB	32.69MiB	37.98MiB	38.32MiB	38.41MiB	38.49MiB	38.66MiB	-8.15123
('datadog_agent_remap_datadog_logs_acks', 'baseline', '6285f4eb-dbe2-4478-9ac9-f331c6dc22c5')	199	37.81MiB	270.75KiB	36.84MiB	37.82MiB	38.18MiB	38.23MiB	38.35MiB	38.4MiB	-0.303747
('datadog_agent_remap_datadog_logs', 'baseline', 'e9388cbd-5d08-467f-ba5c-fb07266c8ee0')	200	37.66MiB	224.04KiB	37.08MiB	37.64MiB	37.96MiB	38.01MiB	38.21MiB	38.26MiB	0.278345
('datadog_agent_remap_datadog_logs', 'comparison', '3a7d76c0-c21d-4849-b60a-61a8e266f19d')	200	37.57MiB	343.18KiB	36.82MiB	37.56MiB	38.05MiB	38.16MiB	38.27MiB	38.31MiB	0.108025
('datadog_agent_remap_datadog_logs_acks', 'baseline', 'ded37e69-e73b-4d28-a946-810cd92b9f58')	203	36.13MiB	221.35KiB	35.61MiB	36.13MiB	36.4MiB	36.49MiB	36.67MiB	36.78MiB	0.150198
('datadog_agent_remap_datadog_logs', 'baseline', '472e0d5a-4a3e-412f-ad02-b72545a6df8b')	200	35.9MiB	240.55KiB	35.21MiB	35.91MiB	36.19MiB	36.25MiB	36.48MiB	36.63MiB	-0.0609263
('datadog_agent_remap_datadog_logs_acks', 'baseline', '678eb197-a497-4bb2-bbd3-a9674ad6f669')	199	35.82MiB	260.91KiB	35.03MiB	35.83MiB	36.13MiB	36.18MiB	36.32MiB	36.44MiB	-0.333782
('datadog_agent_remap_datadog_logs_acks', 'comparison', '180e58b4-51d7-4622-910f-d1b44ecda835')	203	35.79MiB	257.91KiB	35.24MiB	35.78MiB	36.13MiB	36.22MiB	36.37MiB	36.44MiB	0.138607
('datadog_agent_remap_datadog_logs_acks', 'comparison', '8eaea040-494e-4467-8bc8-12a70ac91925')	202	35.63MiB	243.18KiB	34.97MiB	35.64MiB	35.94MiB	36.03MiB	36.13MiB	36.22MiB	-0.0959899
('datadog_agent_remap_datadog_logs', 'baseline', 'fc87ae93-c6f6-4d12-bcb1-5c9261c83769')	200	35.25MiB	697.8KiB	34.12MiB	35.07MiB	36.26MiB	36.44MiB	36.62MiB	36.7MiB	0.471826
('datadog_agent_remap_datadog_logs_acks', 'comparison', 'a2ec2666-333b-491d-940e-fc1e1c04c262')	203	35.22MiB	456.16KiB	29.76MiB	35.28MiB	35.55MiB	35.62MiB	35.72MiB	35.79MiB	-9.20623
('datadog_agent_remap_blackhole_acks', 'baseline', 'bec3ab3a-0b7d-422e-a833-f36dd76931db')	201	28.91MiB	593.27KiB	24.56MiB	28.93MiB	29.61MiB	29.68MiB	29.93MiB	30.36MiB	-2.10971
('datadog_agent_remap_blackhole_acks', 'baseline', 'b5d168e6-70d5-4edb-bc4d-e1ebbbe317e1')	199	28.42MiB	346.32KiB	27.56MiB	28.41MiB	28.85MiB	29.02MiB	29.15MiB	29.25MiB	0.100074
('datadog_agent_remap_blackhole_acks', 'baseline', 'a3295308-41fa-4b6c-bb7c-54bf3d366799')	200	28.14MiB	394.84KiB	27.24MiB	28.12MiB	28.7MiB	28.79MiB	28.92MiB	29.14MiB	0.0852644
('datadog_agent_remap_blackhole_acks', 'comparison', 'f7be98b6-536e-4dd9-846d-22a02bb34b3c')	200	28.07MiB	585.09KiB	26.63MiB	28.06MiB	28.83MiB	28.96MiB	29.41MiB	29.45MiB	-0.000204945
('datadog_agent_remap_blackhole', 'comparison', '3c2bd10c-557b-49db-8586-fa9610070488')	201	27.41MiB	729.09KiB	25.57MiB	27.41MiB	28.24MiB	28.55MiB	28.93MiB	29.58MiB	-0.0803787
('datadog_agent_remap_blackhole', 'comparison', '657cd12e-b5a9-4435-a58a-62c4fb7abd77')	201	27.33MiB	678.43KiB	25.62MiB	27.26MiB	28.19MiB	28.43MiB	28.78MiB	28.91MiB	0.0141515
('datadog_agent_remap_blackhole', 'baseline', 'f76d3c6d-7e1f-4aa3-b38f-05d489f9f148')	202	27.1MiB	903.07KiB	24.5MiB	27.06MiB	28.18MiB	28.42MiB	28.85MiB	29.44MiB	-0.18112
('datadog_agent_remap_blackhole_acks', 'comparison', 'cedd9e08-7c0e-4203-925b-193513fd894e')	200	27.08MiB	536.46KiB	25.71MiB	27.11MiB	27.69MiB	27.84MiB	28.24MiB	28.72MiB	-0.239884
('datadog_agent_remap_blackhole', 'baseline', '385b51ac-0434-49e7-9edd-10e7d2d043e9')	202	26.75MiB	886.49KiB	24.71MiB	26.74MiB	27.84MiB	28.14MiB	28.8MiB	29.04MiB	0.0974764
('datadog_agent_remap_blackhole', 'comparison', '20ad11a8-bc13-4f32-b2df-d3e31a71de44')	201	26.59MiB	716.15KiB	24.02MiB	26.61MiB	27.52MiB	27.83MiB	28.0MiB	28.28MiB	-0.159974
('datadog_agent_remap_blackhole_acks', 'comparison', 'e2f6f51e-4b36-4063-aefb-4af0f4231033')	200	25.5MiB	758.31KiB	23.83MiB	25.44MiB	26.58MiB	26.74MiB	27.03MiB	27.32MiB	0.143012
('datadog_agent_remap_blackhole', 'baseline', '3202a64e-54ac-49aa-a998-04d44021ce7c')	202	24.81MiB	738.67KiB	22.74MiB	24.93MiB	25.56MiB	25.92MiB	26.41MiB	26.47MiB	-0.315419
('http_to_http_noack', 'baseline', 'c380c25a-0ccb-409a-b69e-1c745d9c1bbe')	199	21.79MiB	889.54KiB	19.72MiB	21.74MiB	22.93MiB	23.35MiB	23.64MiB	23.98MiB	0.0949412
('http_to_http_noack', 'baseline', 'e79bb083-20b5-4cac-974d-2d18d9f3afff')	199	21.67MiB	827.5KiB	19.78MiB	21.58MiB	22.62MiB	23.14MiB	23.88MiB	24.45MiB	0.571
('http_to_http_noack', 'comparison', 'abc7b552-3c3e-4038-9f6d-a1d798d5d876')	199	21.66MiB	753.77KiB	19.97MiB	21.62MiB	22.63MiB	22.9MiB	23.26MiB	23.62MiB	0.136944
('http_to_http_noack', 'baseline', '83a87a10-2789-494a-bc81-15a774c11933')	199	21.66MiB	869.43KiB	19.82MiB	21.54MiB	22.86MiB	23.07MiB	23.62MiB	23.77MiB	0.237795
('http_to_http_noack', 'comparison', 'a92e9fa1-bf4c-4fa4-8a93-bdbd530225f1')	199	20.81MiB	931.87KiB	18.91MiB	20.79MiB	22.04MiB	22.33MiB	22.77MiB	23.25MiB	0.134164
('http_to_http_noack', 'comparison', '44ce50b5-12d1-48ae-882b-54bebd6c47a8')	199	20.55MiB	881.38KiB	18.03MiB	20.55MiB	21.72MiB	22.06MiB	22.47MiB	23.35MiB	0.217608
('splunk_hec_indexer_ack_blackhole', 'comparison', 'ce41870a-dc4c-4fe3-a3dd-1e9a8b623677')	200	19.87MiB	745.93KiB	17.92MiB	19.87MiB	20.74MiB	21.01MiB	21.44MiB	21.98MiB	-0.164111
('splunk_hec_indexer_ack_blackhole', 'comparison', '795e5b66-83f1-44a4-8977-8c24740c917a')	200	19.79MiB	792.61KiB	17.64MiB	19.81MiB	20.69MiB	20.95MiB	21.48MiB	22.24MiB	0.00391262
('splunk_hec_indexer_ack_blackhole', 'comparison', 'db776776-ff4f-4241-a23c-7f56e838553b')	200	19.66MiB	761.78KiB	17.42MiB	19.73MiB	20.46MiB	20.82MiB	21.38MiB	21.71MiB	-0.212902
('splunk_hec_indexer_ack_blackhole', 'baseline', 'd8ee3390-d8a2-4de2-b135-34f58b10ca49')	200	19.57MiB	773.23KiB	16.48MiB	19.59MiB	20.43MiB	20.62MiB	21.35MiB	21.96MiB	-0.321833
('splunk_hec_indexer_ack_blackhole', 'baseline', 'b90b0a8d-8073-4861-bbe1-dcee0425660b')	200	19.35MiB	742.64KiB	17.16MiB	19.38MiB	20.2MiB	20.42MiB	20.9MiB	21.04MiB	-0.347023
('splunk_hec_indexer_ack_blackhole', 'baseline', 'df6f657a-9c35-4b1d-913b-c4320e30938e')	200	19.11MiB	961.0KiB	15.99MiB	19.15MiB	20.26MiB	20.53MiB	20.87MiB	21.3MiB	-0.521219
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', '9112aa60-90f5-4849-ba12-9f691657a308')	199	18.35MiB	822.16KiB	16.13MiB	18.41MiB	19.33MiB	19.47MiB	19.92MiB	20.14MiB	-0.395975
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', '07331bcd-cb2f-49cd-95ba-c22b89fac0ee')	200	17.91MiB	765.57KiB	16.27MiB	17.98MiB	18.81MiB	19.01MiB	19.21MiB	19.65MiB	-0.298388
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', 'f6dd243a-d235-43e1-b7c5-51decfd884f8')	200	17.85MiB	843.57KiB	15.43MiB	17.92MiB	18.86MiB	19.01MiB	19.38MiB	19.49MiB	-0.491709
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', '276b80cb-e104-4f80-9c8b-62ea9a19aadc')	199	17.66MiB	851.15KiB	15.64MiB	17.74MiB	18.73MiB	18.85MiB	19.25MiB	19.44MiB	-0.33464
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', 'f18b75b9-9cb9-4ec8-bfa5-4473a90f5cb5')	200	17.39MiB	762.26KiB	15.4MiB	17.43MiB	18.34MiB	18.51MiB	18.74MiB	19.25MiB	-0.197399
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', '0d74f96e-ffee-4476-8b60-de3027aae1fc')	199	17.34MiB	710.29KiB	15.46MiB	17.33MiB	18.2MiB	18.56MiB	18.8MiB	18.89MiB	-0.122134
('splunk_hec_to_splunk_hec_logs_noack', 'baseline', 'c6ad5d03-f04b-4fc9-be72-f39d4892f3d5')	200	17.18MiB	761.29KiB	15.12MiB	17.14MiB	18.09MiB	18.48MiB	18.77MiB	19.16MiB	0.000467631
('splunk_hec_to_splunk_hec_logs_noack', 'comparison', '9f09907f-0e6e-4e37-ba6e-6a3503086a0d')	199	17.17MiB	753.67KiB	15.33MiB	17.22MiB	18.12MiB	18.47MiB	18.8MiB	18.86MiB	0.033809
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', '7ad76836-2c5f-464b-9407-be78d7f98f32')	198	17.14MiB	855.93KiB	14.84MiB	17.27MiB	18.18MiB	18.53MiB	18.71MiB	18.95MiB	-0.24136
('splunk_hec_to_splunk_hec_logs_acks', 'comparison', '7abed7fb-fa2a-4d4e-a8d7-5854d5361500')	198	17.06MiB	855.22KiB	14.36MiB	17.07MiB	18.12MiB	18.39MiB	18.85MiB	18.91MiB	-0.0420874
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', '356bd36c-4da5-4af1-b1e4-cdfcadcce065')	200	16.99MiB	827.81KiB	14.87MiB	16.98MiB	18.03MiB	18.28MiB	18.67MiB	19.52MiB	-0.0335374
('splunk_hec_to_splunk_hec_logs_acks', 'baseline', 'f0da3705-9ff5-4672-be50-c78e70b552a0')	199	16.74MiB	922.33KiB	14.06MiB	16.82MiB	17.88MiB	18.3MiB	18.54MiB	18.62MiB	-0.27882
('http_to_http_acks', 'comparison', 'dfbe50cd-e841-4235-8927-43211a6dc436')	199	10.76MiB	4.93MiB	2.25MiB	10.32MiB	17.77MiB	18.88MiB	19.73MiB	20.05MiB	0.158562
('http_to_http_acks', 'comparison', '4b32fa38-7d02-4165-b025-5af8086a5f69')	199	10.51MiB	4.9MiB	1.34MiB	10.4MiB	17.75MiB	19.78MiB	20.6MiB	21.37MiB	0.341495
('syslog_splunk_hec_logs', 'baseline', '2e881c79-48b7-4eaa-bc50-08a016a813af')	199	10.19MiB	40.69KiB	10.08MiB	10.19MiB	10.24MiB	10.25MiB	10.27MiB	10.29MiB	-0.105265
('syslog_humio_logs', 'comparison', '25f78c99-00ed-4aeb-84a1-cc5104d12362')	199	10.19MiB	30.31KiB	10.11MiB	10.19MiB	10.23MiB	10.24MiB	10.25MiB	10.26MiB	-0.0772524
('syslog_humio_logs', 'comparison', 'b9ced176-78f9-4549-8f43-b625ce1b25fe')	199	10.15MiB	43.95KiB	10.01MiB	10.16MiB	10.2MiB	10.22MiB	10.23MiB	10.26MiB	-0.458335
('syslog_splunk_hec_logs', 'baseline', '271c8324-6e63-4632-a2b5-ffd72679fced')	199	10.12MiB	29.11KiB	10.05MiB	10.12MiB	10.16MiB	10.17MiB	10.18MiB	10.23MiB	0.195935
('syslog_splunk_hec_logs', 'comparison', '2e9c5e82-1124-424e-b02f-5061a6305aa5')	200	10.12MiB	36.86KiB	10.0MiB	10.12MiB	10.16MiB	10.18MiB	10.19MiB	10.21MiB	-0.25129
('syslog_humio_logs', 'baseline', '59841c72-e544-4a30-bb48-859d3f8990e4')	202	10.08MiB	31.54KiB	9.96MiB	10.07MiB	10.11MiB	10.12MiB	10.14MiB	10.16MiB	-0.184623
('syslog_splunk_hec_logs', 'comparison', '0d719c52-5e68-4823-9b89-f7d7b2d170e6')	200	10.03MiB	35.44KiB	9.94MiB	10.03MiB	10.07MiB	10.09MiB	10.1MiB	10.11MiB	-0.287539
('syslog_splunk_hec_logs', 'baseline', 'b053811e-34e0-4200-b2a3-b8a2168f17be')	199	9.99MiB	29.31KiB	9.92MiB	9.99MiB	10.03MiB	10.04MiB	10.05MiB	10.06MiB	-0.0889085
('http_to_http_acks', 'baseline', '474c918b-33d1-4cf9-b23d-29f5f86a4358')	200	9.92MiB	4.49MiB	2.07MiB	9.93MiB	17.95MiB	18.94MiB	20.14MiB	20.21MiB	0.464542
('syslog_log2metric_splunk_hec_metrics', 'comparison', 'b5690b75-7f2f-467b-bb93-6409f94c43d1')	199	9.78MiB	32.63KiB	9.69MiB	9.78MiB	9.82MiB	9.84MiB	9.85MiB	9.88MiB	0.160998
('syslog_humio_logs', 'comparison', 'f3c405a0-0208-406b-9ccf-b0d86487e6c6')	72	9.73MiB	1.78MiB	150.22KiB	10.11MiB	10.15MiB	10.16MiB	10.18MiB	10.2MiB	-4.95006
('syslog_splunk_hec_logs', 'comparison', '429120a7-35d9-443e-9154-f0ccfc8e4d04')	82	9.6MiB	1.63MiB	24.92KiB	9.91MiB	9.95MiB	9.96MiB	9.99MiB	10.0MiB	-5.47601
('http_to_http_acks', 'comparison', '477c9dd0-f887-44c0-8ff5-ea597272409b')	199	9.55MiB	4.2MiB	2.41MiB	9.44MiB	15.47MiB	17.65MiB	18.86MiB	20.46MiB	0.358134
('syslog_humio_logs', 'baseline', 'cdd45e07-6daa-4aa1-9da8-d65165867516')	202	9.54MiB	29.06KiB	9.46MiB	9.54MiB	9.57MiB	9.58MiB	9.6MiB	9.62MiB	-0.188609
('syslog_humio_logs', 'baseline', '836f4532-0671-44f4-abc9-cbfa78dbbdac')	202	9.53MiB	45.18KiB	9.44MiB	9.54MiB	9.59MiB	9.61MiB	9.63MiB	9.63MiB	0.0791102
('syslog_log2metric_splunk_hec_metrics', 'comparison', '3518f47d-19fd-4031-a58d-9663b55c8140')	200	9.53MiB	46.1KiB	9.33MiB	9.54MiB	9.57MiB	9.6MiB	9.61MiB	9.61MiB	-1.00339
('syslog_log2metric_splunk_hec_metrics', 'comparison', '5c519d84-93e9-4bae-990f-c1ae774d6b9f')	200	9.42MiB	24.39KiB	9.36MiB	9.42MiB	9.45MiB	9.46MiB	9.46MiB	9.49MiB	-0.100681
('syslog_log2metric_splunk_hec_metrics', 'baseline', '9e85e472-9127-47ff-a998-d946252e89be')	200	9.31MiB	29.33KiB	9.24MiB	9.31MiB	9.34MiB	9.36MiB	9.37MiB	9.41MiB	0.186374
('syslog_log2metric_splunk_hec_metrics', 'baseline', 'cfb05924-61c0-4acc-98f0-6a6cf40dfcda')	200	9.29MiB	26.39KiB	9.2MiB	9.29MiB	9.33MiB	9.33MiB	9.35MiB	9.37MiB	-0.080359
('syslog_log2metric_splunk_hec_metrics', 'baseline', '4215776b-d7d1-4690-b741-551f8402159d')	200	9.24MiB	37.54KiB	9.12MiB	9.24MiB	9.29MiB	9.3MiB	9.31MiB	9.33MiB	-0.10941
('http_to_http_acks', 'baseline', '0085a279-26e3-42ec-a33b-e7a4a560c022')	200	9.19MiB	3.94MiB	1.34MiB	9.32MiB	15.0MiB	16.74MiB	19.9MiB	20.06MiB	0.624545
('http_to_http_acks', 'baseline', '5e1ae1fa-5223-46df-81f8-1e2c0aba3443')	200	8.87MiB	3.69MiB	1.34MiB	9.26MiB	12.65MiB	16.61MiB	18.55MiB	19.48MiB	0.54604
('splunk_hec_route_s3', 'baseline', 'fb38ce5d-6d2c-48f8-9776-e8b268ed4f93')	199	8.74MiB	2.03MiB	4.83MiB	8.49MiB	11.48MiB	12.04MiB	12.98MiB	13.89MiB	0.278304
('splunk_hec_route_s3', 'baseline', 'cf701682-a803-4032-8a1e-bc540eeca34a')	199	8.74MiB	1.98MiB	5.04MiB	8.7MiB	11.27MiB	11.98MiB	13.08MiB	14.32MiB	0.23188
('splunk_hec_route_s3', 'comparison', '70727b7a-2608-4aca-b281-75ff17f565e1')	199	8.7MiB	1.95MiB	4.48MiB	8.82MiB	11.31MiB	11.58MiB	12.22MiB	12.52MiB	-0.0548005
('splunk_hec_route_s3', 'comparison', 'a26cf840-55be-4b90-80b1-a75988c8dc13')	199	8.69MiB	2.04MiB	4.17MiB	8.8MiB	11.29MiB	11.72MiB	12.96MiB	13.22MiB	0.0452999
('splunk_hec_route_s3', 'baseline', 'a1a83623-e9e3-4320-9700-21f90729e459')	200	8.67MiB	2.06MiB	4.8MiB	8.66MiB	11.35MiB	11.75MiB	13.08MiB	13.48MiB	0.130516
('splunk_hec_route_s3', 'comparison', '29152556-8195-4f8d-b76b-727193bd8477')	199	8.64MiB	1.82MiB	4.89MiB	8.82MiB	10.97MiB	11.53MiB	12.5MiB	12.96MiB	0.0383606
('syslog_log2metric_humio_metrics', 'comparison', '1a40151f-f7ac-4f49-b1f2-6fa9a894de9f')	200	8.1MiB	32.18KiB	8.0MiB	8.1MiB	8.14MiB	8.15MiB	8.16MiB	8.17MiB	-0.263527
('syslog_log2metric_humio_metrics', 'comparison', '112c560d-a986-4225-af07-5e2d54dd6218')	200	7.93MiB	34.34KiB	7.83MiB	7.94MiB	7.97MiB	7.99MiB	8.0MiB	8.02MiB	-0.194378
('syslog_log2metric_humio_metrics', 'comparison', '2b1b95a9-264c-47f8-82f7-d3d3fb98f636')	73	7.82MiB	1.57MiB	12.97KiB	8.17MiB	8.22MiB	8.23MiB	8.28MiB	8.29MiB	-4.51757
('syslog_log2metric_humio_metrics', 'baseline', '70cf6b83-a58b-4745-a895-96fbeeedc8cb')	202	7.77MiB	35.49KiB	7.68MiB	7.77MiB	7.82MiB	7.83MiB	7.85MiB	7.85MiB	0.0665739
('syslog_regex_logs2metric_ddmetrics', 'comparison', '1484c1dc-a70e-4b0f-ad97-f248bd84359d')	200	7.76MiB	282.35KiB	6.77MiB	7.65MiB	8.18MiB	8.23MiB	8.27MiB	8.35MiB	0.344949
('syslog_regex_logs2metric_ddmetrics', 'comparison', '5bd3e07e-6347-4341-a946-cc0f4e312902')	200	7.61MiB	314.84KiB	6.7MiB	7.5MiB	8.05MiB	8.09MiB	8.11MiB	8.17MiB	0.258778
('syslog_regex_logs2metric_ddmetrics', 'comparison', '7d5ed20e-7588-4280-8ab3-0992b9a0055c')	200	7.51MiB	321.91KiB	7.07MiB	7.47MiB	7.93MiB	7.97MiB	8.01MiB	8.04MiB	0.221126
('syslog_log2metric_humio_metrics', 'baseline', '119b7abb-0f05-40ae-9245-ecd08270a113')	202	7.42MiB	33.12KiB	7.32MiB	7.42MiB	7.45MiB	7.46MiB	7.49MiB	7.5MiB	-0.106455
('syslog_regex_logs2metric_ddmetrics', 'baseline', '94595397-ccde-4ea3-b0a6-4b6bcf34c02a')	200	7.39MiB	311.72KiB	6.56MiB	7.37MiB	7.77MiB	7.78MiB	7.83MiB	7.84MiB	0.0111894
('syslog_regex_logs2metric_ddmetrics', 'baseline', '18b7ad4c-f8c5-4ea8-b29c-1a212ea97add')	200	7.33MiB	319.25KiB	6.87MiB	7.34MiB	7.71MiB	7.72MiB	7.74MiB	7.78MiB	0.00536307
('syslog_regex_logs2metric_ddmetrics', 'baseline', '6ce13194-bc68-4eeb-aa02-626882027a0f')	200	7.23MiB	316.66KiB	6.77MiB	7.25MiB	7.59MiB	7.62MiB	7.66MiB	7.69MiB	-0.0616864
('syslog_log2metric_humio_metrics', 'baseline', '25a07122-5bab-48bb-9083-929c3e5a648f')	75	7.14MiB	1.81MiB	12.75KiB	7.6MiB	7.82MiB	7.82MiB	7.84MiB	7.86MiB	-3.57292
('syslog_loki', 'baseline', 'a6d9f7e1-4fa3-4a03-882a-8d4cc5338e07')	202	7.08MiB	56.16KiB	6.89MiB	7.09MiB	7.14MiB	7.16MiB	7.2MiB	7.22MiB	-0.371378
('syslog_loki', 'baseline', '95195dae-6ad4-4ed3-9261-aa1399650111')	202	6.76MiB	35.56KiB	6.66MiB	6.76MiB	6.81MiB	6.81MiB	6.83MiB	6.83MiB	-0.411707
('syslog_loki', 'comparison', 'd282bc87-1448-43fc-a11f-6d969c62356b')	200	6.73MiB	42.09KiB	6.6MiB	6.74MiB	6.78MiB	6.8MiB	6.81MiB	6.82MiB	-0.869981
('syslog_loki', 'baseline', '6e6bc049-8b0e-41b3-9f5b-2e45faf31dc5')	203	6.73MiB	55.46KiB	6.56MiB	6.74MiB	6.8MiB	6.82MiB	6.83MiB	6.84MiB	-0.423565
('syslog_loki', 'comparison', '25a1c793-5816-4fc6-89c1-85ce53daafd8')	200	6.64MiB	288.35KiB	6.05MiB	6.8MiB	6.87MiB	6.88MiB	6.91MiB	6.92MiB	-1.04042
('syslog_loki', 'comparison', '3cf2e56b-c80d-42f8-b45e-5991d130f9bb')	200	6.53MiB	303.93KiB	6.08MiB	6.66MiB	6.85MiB	6.86MiB	6.87MiB	6.88MiB	-0.27329
('splunk_transforms_splunk3', 'baseline', 'd64a5078-1294-406b-8c25-180ba1522e6e')	203	5.51MiB	1.55MiB	2.58MiB	5.15MiB	7.61MiB	8.04MiB	8.63MiB	8.82MiB	0.326932
('splunk_transforms_splunk3', 'comparison', '95ab5c72-b226-44df-96af-e95c54562194')	202	5.47MiB	1.52MiB	2.99MiB	5.21MiB	7.55MiB	7.97MiB	8.46MiB	9.48MiB	0.337097
('splunk_transforms_splunk3', 'comparison', 'a4e60531-6c71-45ea-98dc-4caba39bb721')	202	5.27MiB	1.44MiB	2.82MiB	5.01MiB	7.31MiB	7.81MiB	8.22MiB	8.37MiB	0.286825
('splunk_transforms_splunk3', 'comparison', '257886ed-2fec-45da-80ff-621c295c2b67')	202	5.26MiB	1.45MiB	2.53MiB	5.02MiB	7.22MiB	7.66MiB	8.18MiB	8.94MiB	0.277734
('splunk_transforms_splunk3', 'baseline', '2bfd4df2-7c1e-4dfc-b16b-de8048a62451')	202	5.26MiB	1.54MiB	2.44MiB	4.88MiB	7.45MiB	7.79MiB	8.52MiB	8.76MiB	0.35751
('splunk_transforms_splunk3', 'baseline', '3f6ac2bf-0b42-41f9-84ab-f31288af5315')	203	5.21MiB	1.44MiB	2.17MiB	5.07MiB	7.22MiB	7.45MiB	8.16MiB	8.31MiB	0.203318
('http_datadog_filter_blackhole', 'baseline', '8899a55f-7656-4b20-946f-8e6204c54c9a')	202	2.27MiB	796.79KiB	521.99KiB	2.21MiB	3.32MiB	3.73MiB	4.0MiB	4.17MiB	0.27506
('http_datadog_filter_blackhole', 'baseline', '7a00816f-f899-4ed4-a097-79d1063f308c')	202	2.25MiB	802.7KiB	506.0KiB	2.22MiB	3.35MiB	3.75MiB	3.85MiB	4.05MiB	0.156468
('http_datadog_filter_blackhole', 'baseline', '39f57a69-69a9-465a-8646-c2296900c98e')	203	2.23MiB	881.45KiB	292.97KiB	2.13MiB	3.36MiB	3.73MiB	4.2MiB	4.49MiB	0.284338
('http_datadog_filter_blackhole', 'comparison', '78484926-58a5-4703-a17c-eedb7d8ad55d')	202	2.19MiB	844.56KiB	506.0KiB	2.14MiB	3.24MiB	3.51MiB	4.21MiB	4.61MiB	0.288598
('http_datadog_filter_blackhole', 'comparison', 'bc8afbf0-d18c-4d0b-a8cd-617fd794c1c3')	202	2.14MiB	788.01KiB	490.01KiB	2.03MiB	3.1MiB	3.43MiB	3.9MiB	4.3MiB	0.268565
('http_datadog_filter_blackhole', 'comparison', 'ed1bf2db-f214-4c5a-8309-710b9f0d274e')	202	2.13MiB	803.75KiB	400.0KiB	2.05MiB	3.18MiB	3.36MiB	3.85MiB	4.49MiB	0.159689
('http_pipelines_no_grok_blackhole', 'baseline', '78f85bf4-818e-4431-9043-da97cb6f4769')	200	1.99MiB	835.19KiB	245.0KiB	1.9MiB	3.15MiB	3.51MiB	3.81MiB	4.19MiB	0.279244
('http_pipelines_no_grok_blackhole', 'comparison', 'db61fa20-a9ab-4c9e-bba4-cf93213ca36f')	200	1.99MiB	806.52KiB	260.99KiB	1.87MiB	3.11MiB	3.51MiB	3.83MiB	4.02MiB	0.366186
('http_pipelines_no_grok_blackhole', 'comparison', 'af23d788-1386-4ef1-af34-4a8f5832c7be')	200	1.99MiB	793.89KiB	260.99KiB	1.87MiB	3.14MiB	3.34MiB	3.72MiB	4.02MiB	0.341223
('http_pipelines_no_grok_blackhole', 'comparison', 'f9a2ab27-1f3f-4c69-b6ac-433ea876edc0')	200	1.95MiB	838.15KiB	276.98KiB	1.78MiB	3.14MiB	3.51MiB	3.76MiB	4.19MiB	0.389987
('http_pipelines_no_grok_blackhole', 'baseline', 'a57a224d-fbeb-4570-b1bb-7b2e8c32827d')	200	1.86MiB	776.06KiB	490.01KiB	1.76MiB	2.8MiB	3.2MiB	3.51MiB	3.55MiB	0.227888
('http_pipelines_no_grok_blackhole', 'baseline', '51b4d457-4c25-499f-a5c5-53353176d17b')	200	1.86MiB	783.52KiB	186.49KiB	1.76MiB	2.84MiB	3.2MiB	3.7MiB	3.81MiB	0.308171
('http_pipelines_blackhole_acks', 'comparison', '5098764d-f392-48eb-97ec-26cfffe55bc8')	199	884.51KiB	512.33KiB	31.98KiB	798.87KiB	1.6MiB	1.7MiB	1.99MiB	2.05MiB	0.357071
('http_pipelines_blackhole_acks', 'baseline', '8a1735ed-c4c9-45cc-a063-8eb48bfceeb2')	200	881.47KiB	527.03KiB	31.98KiB	813.82KiB	1.59MiB	1.72MiB	2.26MiB	2.53MiB	0.645388
('http_pipelines_blackhole_acks', 'comparison', 'ff0202e2-f3c4-421a-bb2e-51e1e9b751ed')	200	859.94KiB	501.17KiB	31.98KiB	797.83KiB	1.47MiB	1.63MiB	2.11MiB	2.5MiB	0.66142
('http_pipelines_blackhole', 'baseline', '533eb185-6753-4b2f-b0af-cbc03480cc99')	197	847.28KiB	499.91KiB	31.98KiB	751.47KiB	1.57MiB	1.66MiB	2.08MiB	2.17MiB	0.66345
('http_pipelines_blackhole', 'comparison', 'baa313cb-50a7-4178-8963-6dad8efbc49e')	197	817.73KiB	500.38KiB	31.98KiB	721.1KiB	1.56MiB	1.64MiB	2.0MiB	2.2MiB	0.666159
('http_pipelines_blackhole', 'comparison', 'fbe7c32f-6132-4a24-91c4-ca89e7f9d022')	200	813.35KiB	496.65KiB	15.99KiB	667.31KiB	1.57MiB	1.63MiB	1.96MiB	2.06MiB	0.652
('http_pipelines_blackhole_acks', 'comparison', '40829238-4524-4431-9479-e851418b2f0a')	198	802.83KiB	558.94KiB	15.99KiB	690.82KiB	1.48MiB	1.6MiB	2.5MiB	2.9MiB	1.10404
('http_pipelines_blackhole', 'baseline', 'feacda40-19df-468a-83e4-eeb8b461d363')	197	792.15KiB	477.24KiB	15.99KiB	675.38KiB	1.5MiB	1.6MiB	1.96MiB	2.09MiB	0.589577
('http_pipelines_blackhole', 'baseline', '4a1304f7-1b23-4727-be4c-17a0ea71756f')	198	790.53KiB	492.74KiB	15.99KiB	667.31KiB	1.56MiB	1.63MiB	2.06MiB	2.31MiB	0.85974
('http_pipelines_blackhole', 'comparison', '779d9d9d-c15b-4553-a3a9-a8a676cfc986')	199	784.09KiB	479.27KiB	31.98KiB	675.38KiB	1.47MiB	1.62MiB	2.1MiB	2.34MiB	0.877912
('http_pipelines_blackhole_acks', 'baseline', '51e159be-561f-4ed3-9d58-e281125cd7b2')	200	783.61KiB	481.11KiB	31.98KiB	697.98KiB	1.39MiB	1.61MiB	2.12MiB	2.38MiB	0.758568
('http_pipelines_blackhole_acks', 'baseline', '4230acbd-bf7b-40a6-8e0c-d87558486052')	198	775.03KiB	496.31KiB	30.89KiB	705.43KiB	1.53MiB	1.66MiB	1.93MiB	2.06MiB	0.570828

Jan 17 '22 20:01 github-actions[bot]

Ok, so this works for Linux (and macOS apparently), but fails to link on ARM64 and Windows due to openssl symbol issues. This is undoubtedly due to the update to the rdkafka crate, but that's required to fix a crash issues that this would otherwise trigger. Any suggestions for how to resolve this?

Jan 18 '22 21:01 bruceg

Ah, yeah, seemingly those are the same failures we saw on https://github.com/vectordotdev/vector/pull/10302. I think someone will need to dig in to see what we need to do to upgrade. Do you want to do that? Or we could tap someone else?

Jan 18 '22 21:01 jszwedko

Just noting that this is still unfortunately blocked by the rdkafka upgrade 😢 https://github.com/vectordotdev/vector/pull/10302

Feb 04 '22 17:02 jszwedko

Hello Guys, I just lost some events on production due to this issue. It means that I'm eager to see it in the next release :)

Feb 09 '22 12:02 udleinati

Hi @udleinati ! That sucks, I'm sorry. Unfortunately this PR is still blocked be cause we are running into issues upgrading rdkafka, but we hope to get it unblocked soon. https://github.com/vectordotdev/vector/pull/10302

Feb 09 '22 15:02 jszwedko

Soak Test Results

Baseline: db38b570a2271719e35533a601b0960abfa44c32 Comparison: eae36b646ec5711d331ef3338cea936c79a75e0e Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	40.68KiB	3.1	98.67%	1.28MiB	681.44KiB	11.62KiB	0.518473	1.32MiB	680.17KiB	11.61KiB	0.501967	True	True
splunk_transforms_splunk3	113.69KiB	0.93	97.09%	11.98MiB	2.13MiB	36.63KiB	0.178069	12.09MiB	2.14MiB	37.03KiB	0.176818	False	False
datadog_agent_remap_blackhole	433.94KiB	0.89	100.00%	47.73MiB	2.34MiB	40.93KiB	0.0490078	48.15MiB	2.35MiB	41.15KiB	0.0487988	False	False
datadog_agent_remap_blackhole_acks	427.88KiB	0.82	100.00%	50.79MiB	1.27MiB	21.77KiB	0.0249635	51.2MiB	1.12MiB	19.58KiB	0.0219479	False	False
syslog_log2metric_humio_metrics	107.55KiB	0.78	100.00%	13.43MiB	181.59KiB	3.04KiB	0.0131997	13.54MiB	191.41KiB	3.2KiB	0.0138057	False	False
syslog_humio_logs	89.9KiB	0.58	100.00%	15.26MiB	672.43KiB	11.44KiB	0.0430213	15.35MiB	674.96KiB	11.27KiB	0.0429361	False	False
http_datadog_filter_blackhole	36.75KiB	0.25	94.92%	14.57MiB	781.87KiB	13.36KiB	0.0523837	14.61MiB	774.76KiB	13.25KiB	0.0517797	False	True
syslog_regex_logs2metric_ddmetrics	28.85KiB	0.24	100.00%	11.98MiB	235.43KiB	3.94KiB	0.0191811	12.01MiB	255.2KiB	4.35KiB	0.0207429	False	False
syslog_loki	31.85KiB	0.21	99.94%	14.9MiB	349.48KiB	5.85KiB	0.0228946	14.94MiB	428.91KiB	7.27KiB	0.0280398	False	False
http_pipelines_no_grok_blackhole	23.53KiB	0.18	77.01%	12.52MiB	828.03KiB	14.16KiB	0.0646004	12.54MiB	792.59KiB	13.55KiB	0.0617223	False	False
fluent_elasticsearch	299.13B	0	18.49%	79.47MiB	53.78KiB	912.16B	0.000660756	79.47MiB	52.66KiB	896.91B	0.000647023	False	False
splunk_hec_to_splunk_hec_logs_acks	-31.27B	-0	0.31%	23.84MiB	329.81KiB	5.51KiB	0.0135092	23.84MiB	333.96KiB	5.57KiB	0.0136792	False	False
splunk_hec_indexer_ack_blackhole	645.42B	0	6.35%	23.84MiB	334.77KiB	5.59KiB	0.0137134	23.84MiB	335.61KiB	5.6KiB	0.0137476	False	False
http_to_http_noack	-2.45KiB	-0.01	32.14%	23.85MiB	247.03KiB	4.17KiB	0.0101145	23.84MiB	247.31KiB	4.18KiB	0.0101271	False	False
splunk_hec_to_splunk_hec_logs_noack	-5.52KiB	-0.02	50.64%	23.84MiB	334.95KiB	5.66KiB	0.0137159	23.84MiB	340.01KiB	5.75KiB	0.0139263	False	False
http_to_http_json	-7.15KiB	-0.03	58.96%	23.84MiB	363.52KiB	6.11KiB	0.0148884	23.83MiB	365.97KiB	6.18KiB	0.0149934	False	False
syslog_splunk_hec_logs	-14.13KiB	-0.09	94.02%	16.06MiB	308.54KiB	5.2KiB	0.0187528	16.05MiB	321.49KiB	5.42KiB	0.0195571	False	False
syslog_log2metric_splunk_hec_metrics	-36.1KiB	-0.22	100.00%	16.15MiB	168.49KiB	2.87KiB	0.0101839	16.12MiB	168.63KiB	2.88KiB	0.0102143	False	False
splunk_hec_route_s3	-49.23KiB	-0.25	63.22%	19.35MiB	2.23MiB	38.63KiB	0.11536	19.3MiB	2.23MiB	38.68KiB	0.115591	False	False
http_to_http_acks	-103.46KiB	-0.55	42.28%	18.41MiB	7.73MiB	133.12KiB	0.419973	18.31MiB	7.47MiB	129.3KiB	0.408012	True	True
socket_to_socket_blackhole	-177.23KiB	-0.73	100.00%	23.83MiB	822.1KiB	13.86KiB	0.0336921	23.65MiB	778.66KiB	13.26KiB	0.0321453	False	False
datadog_agent_remap_datadog_logs	-808.29KiB	-1.14	100.00%	69.5MiB	426.49KiB	7.18KiB	0.00599189	68.71MiB	471.78KiB	8.0KiB	0.00670436	False	False
datadog_agent_remap_datadog_logs_acks	-1.13MiB	-1.68	100.00%	66.91MiB	2.91MiB	50.54KiB	0.0435059	65.78MiB	3.04MiB	52.09KiB	0.0462436	False	False
http_pipelines_blackhole	-23.63KiB	-1.74	87.41%	1.33MiB	641.51KiB	10.79KiB	0.47221	1.3MiB	648.02KiB	11.04KiB	0.485445	True	True

May 02 '22 22:05 github-actions[bot]

Soak Test Results

Baseline: d65f6cc19f5545d8faf1ba497f08dd93d20260f5 Comparison: 7b048e853d667439a67c070d1c120de8393df46b Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_splunk_hec_metrics	514.73KiB	3.13	100.00%	16.08MiB	298.09KiB	6.22KiB	0.0181042	16.58MiB	226.69KiB	4.75KiB	0.0133504	False	False
syslog_regex_logs2metric_ddmetrics	345.72KiB	2.84	100.00%	11.89MiB	288.87KiB	5.92KiB	0.0237218	12.23MiB	386.62KiB	8.07KiB	0.0308723	False	False
syslog_humio_logs	421.87KiB	2.53	100.00%	16.26MiB	157.71KiB	3.28KiB	0.00946734	16.68MiB	127.7KiB	2.61KiB	0.00747684	False	False
syslog_splunk_hec_logs	387.54KiB	2.38	100.00%	15.9MiB	428.85KiB	8.86KiB	0.0263298	16.28MiB	414.6KiB	8.56KiB	0.0248636	False	False
http_to_http_acks	387.22KiB	2.09	90.32%	18.07MiB	8.04MiB	169.44KiB	0.444514	18.45MiB	7.56MiB	160.16KiB	0.409537	True	True
syslog_loki	296.28KiB	1.98	100.00%	14.64MiB	447.22KiB	9.17KiB	0.0298264	14.93MiB	499.82KiB	10.37KiB	0.0326885	False	False
http_pipelines_no_grok_blackhole	62.26KiB	0.48	99.41%	12.6MiB	769.41KiB	16.11KiB	0.0596298	12.66MiB	757.24KiB	15.86KiB	0.0584043	False	False
http_datadog_filter_blackhole	58.27KiB	0.39	98.85%	14.46MiB	778.69KiB	16.29KiB	0.0525684	14.52MiB	778.93KiB	16.31KiB	0.0523784	False	True
syslog_log2metric_humio_metrics	11.93KiB	0.09	61.82%	13.08MiB	481.47KiB	9.87KiB	0.0359528	13.09MiB	458.98KiB	9.41KiB	0.0342426	False	False
splunk_hec_indexer_ack_blackhole	1.98KiB	0.01	16.46%	23.84MiB	328.89KiB	6.73KiB	0.0134711	23.84MiB	331.4KiB	6.78KiB	0.0135728	False	False
splunk_hec_to_splunk_hec_logs_noack	-516.19B	-0	4.03%	23.84MiB	339.54KiB	7.03KiB	0.0139055	23.84MiB	340.93KiB	7.06KiB	0.0139626	False	False
fluent_elasticsearch	336.9B	0	17.15%	79.47MiB	53.3KiB	1.08KiB	0.000654774	79.47MiB	52.28KiB	1.07KiB	0.00064231	False	False
splunk_hec_to_splunk_hec_logs_acks	-647.46B	-0	5.26%	23.84MiB	330.72KiB	6.76KiB	0.0135461	23.84MiB	331.58KiB	6.78KiB	0.0135819	False	False
datadog_agent_remap_datadog_logs_acks	-4.92KiB	-0.01	5.84%	65.43MiB	2.73MiB	58.11KiB	0.0417371	65.43MiB	1.61MiB	33.67KiB	0.0245341	False	False
http_to_http_noack	-4.34KiB	-0.02	44.50%	23.85MiB	251.79KiB	5.2KiB	0.0103087	23.84MiB	250.28KiB	5.19KiB	0.0102485	False	False
http_to_http_json	-6.62KiB	-0.03	47.20%	23.84MiB	360.03KiB	7.4KiB	0.0147439	23.83MiB	359.64KiB	7.43KiB	0.014732	False	False
datadog_agent_remap_blackhole	-37.06KiB	-0.07	44.39%	49.15MiB	2.07MiB	44.45KiB	0.0421668	49.11MiB	2.08MiB	44.57KiB	0.0423059	False	False
http_pipelines_blackhole_acks	-1.4KiB	-0.1	5.52%	1.37MiB	685.73KiB	14.32KiB	0.490456	1.36MiB	677.97KiB	14.17KiB	0.485392	True	True
splunk_hec_route_s3	-66.15KiB	-0.33	69.77%	19.42MiB	2.13MiB	45.05KiB	0.10952	19.36MiB	2.15MiB	45.62KiB	0.110991	False	False
splunk_transforms_splunk3	-63.46KiB	-0.5	68.36%	12.31MiB	2.12MiB	44.51KiB	0.17189	12.25MiB	2.12MiB	45.06KiB	0.173443	False	False
datadog_agent_remap_blackhole_acks	-431.76KiB	-0.82	100.00%	51.51MiB	626.98KiB	12.88KiB	0.011885	51.08MiB	586.84KiB	12.23KiB	0.0112158	False	False
datadog_agent_remap_datadog_logs	-713.65KiB	-1	100.00%	69.52MiB	335.27KiB	6.91KiB	0.00470849	68.82MiB	296.77KiB	6.16KiB	0.00421003	False	False
http_pipelines_blackhole	-20.47KiB	-1.46	71.61%	1.37MiB	648.41KiB	13.36KiB	0.461207	1.35MiB	653.81KiB	13.65KiB	0.471915	True	True
socket_to_socket_blackhole	-861.51KiB	-3.52	100.00%	23.88MiB	465.99KiB	9.62KiB	0.0190489	23.04MiB	471.59KiB	9.84KiB	0.0199816	False	False

May 04 '22 02:05 github-actions[bot]

Soak Test Results

Baseline: b2db4870575ced8e1fe7684581e3706e72ae80d0 Comparison: ea1f389e79189374396b62689c7479ffc085420f Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_to_http_acks	400.53KiB	2.18	91.48%	17.94MiB	7.65MiB	161.39KiB	0.426423	18.33MiB	7.91MiB	167.56KiB	0.431438	True	True
socket_to_socket_blackhole	126.73KiB	0.51	100.00%	24.12MiB	204.67KiB	4.23KiB	0.00828471	24.24MiB	160.17KiB	3.34KiB	0.00645051	False	False
datadog_agent_remap_blackhole	213.69KiB	0.32	100.00%	65.41MiB	1.3MiB	27.85KiB	0.0198491	65.62MiB	972.6KiB	20.37KiB	0.014471	False	False
http_pipelines_blackhole_acks	2.51KiB	0.06	10.22%	3.85MiB	657.75KiB	13.77KiB	0.166802	3.85MiB	664.0KiB	13.91KiB	0.168282	False	True
http_to_http_json	1.3KiB	0.01	10.19%	23.84MiB	349.17KiB	7.19KiB	0.0143024	23.84MiB	347.24KiB	7.18KiB	0.0142225	False	False
fluent_elasticsearch	-33.56B	-0	1.73%	79.47MiB	52.91KiB	1.07KiB	0.000650032	79.47MiB	51.99KiB	1.06KiB	0.000638666	False	False
splunk_hec_to_splunk_hec_logs_acks	999.42B	0	8.02%	23.84MiB	331.47KiB	6.78KiB	0.0135769	23.84MiB	338.77KiB	6.93KiB	0.0138752	False	False
splunk_hec_to_splunk_hec_logs_noack	-275.1B	-0	2.19%	23.84MiB	332.3KiB	6.88KiB	0.0136066	23.84MiB	335.35KiB	6.95KiB	0.0137315	False	False
splunk_hec_indexer_ack_blackhole	726.04B	0	5.90%	23.84MiB	332.44KiB	6.8KiB	0.0136171	23.84MiB	329.75KiB	6.74KiB	0.0135063	False	False
http_to_http_noack	-4.74KiB	-0.02	48.74%	23.85MiB	249.49KiB	5.16KiB	0.0102146	23.84MiB	245.29KiB	5.09KiB	0.0100445	False	False
splunk_hec_route_s3	-18.17KiB	-0.08	23.77%	20.93MiB	1.98MiB	42.0KiB	0.0947257	20.91MiB	2.03MiB	42.96KiB	0.0968404	False	False
datadog_agent_remap_blackhole_acks	-65.72KiB	-0.1	75.28%	64.13MiB	1.92MiB	40.31KiB	0.0298946	64.07MiB	1.87MiB	40.0KiB	0.0292527	False	False
http_pipelines_no_grok_blackhole	-15.06KiB	-0.11	47.13%	13.95MiB	802.02KiB	16.8KiB	0.056128	13.94MiB	811.83KiB	17.0KiB	0.0568743	False	False
syslog_log2metric_splunk_hec_metrics	-34.55KiB	-0.18	100.00%	19.18MiB	211.81KiB	4.42KiB	0.0107808	19.15MiB	197.81KiB	4.14KiB	0.0100862	False	False
http_pipelines_blackhole	-7.78KiB	-0.19	29.70%	3.98MiB	704.51KiB	14.52KiB	0.172908	3.97MiB	687.7KiB	14.34KiB	0.169105	False	True
syslog_splunk_hec_logs	-38.12KiB	-0.21	97.32%	18.13MiB	610.12KiB	12.6KiB	0.0328616	18.09MiB	567.0KiB	11.71KiB	0.0306016	False	False
syslog_humio_logs	-64.16KiB	-0.33	100.00%	18.92MiB	141.86KiB	2.95KiB	0.00732	18.86MiB	150.82KiB	3.08KiB	0.00780812	False	False
syslog_regex_logs2metric_ddmetrics	-51.95KiB	-0.37	99.75%	13.56MiB	580.85KiB	11.91KiB	0.0418223	13.51MiB	595.48KiB	12.42KiB	0.0430364	False	False
http_datadog_filter_blackhole	-105.85KiB	-0.67	100.00%	15.49MiB	715.22KiB	14.96KiB	0.0450737	15.39MiB	777.17KiB	16.28KiB	0.0493065	False	True
syslog_loki	-158.55KiB	-0.91	100.00%	16.94MiB	157.78KiB	3.23KiB	0.00909459	16.78MiB	151.74KiB	3.15KiB	0.00882735	False	False
datadog_agent_remap_datadog_logs	-843.36KiB	-1.07	100.00%	76.61MiB	661.2KiB	13.63KiB	0.00842611	75.79MiB	625.71KiB	12.99KiB	0.00806049	False	False
syslog_log2metric_humio_metrics	-178.23KiB	-1.23	100.00%	14.14MiB	113.99KiB	2.34KiB	0.00787019	13.97MiB	110.95KiB	2.27KiB	0.0077553	False	False
datadog_agent_remap_datadog_logs_acks	-1.02MiB	-1.37	100.00%	74.41MiB	1.02MiB	21.71KiB	0.0137075	73.39MiB	1.73MiB	36.25KiB	0.0235507	False	False
splunk_transforms_splunk3	-281.87KiB	-1.73	100.00%	15.92MiB	2.3MiB	48.49KiB	0.144681	15.65MiB	2.33MiB	49.39KiB	0.14877	False	False

May 11 '22 23:05 github-actions[bot]

Soak Test Results

Baseline: 057df18f1bd617fd6008194b9be36738c370c745 Comparison: a0d8f2afba6831dd032f91e48480324f68f29c07 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
datadog_agent_remap_blackhole	1.66MiB	2.59	100.00%	64.1MiB	4.9MiB	102.14KiB	0.0764371	65.77MiB	3.87MiB	80.72KiB	0.058797	False	False
syslog_log2metric_humio_metrics	169.91KiB	1.22	100.00%	13.56MiB	206.72KiB	4.22KiB	0.0148832	13.73MiB	542.97KiB	11.06KiB	0.0386197	False	False
splunk_hec_route_s3	140.66KiB	0.7	97.66%	19.62MiB	2.15MiB	44.68KiB	0.109365	19.75MiB	2.06MiB	43.05KiB	0.104016	False	False
socket_to_socket_blackhole	41.67KiB	0.3	100.00%	13.68MiB	237.72KiB	4.85KiB	0.0169649	13.72MiB	122.58KiB	2.5KiB	0.00872214	False	False
splunk_hec_to_splunk_hec_logs_noack	10.67KiB	0.04	66.10%	23.83MiB	434.72KiB	8.87KiB	0.0178139	23.84MiB	331.11KiB	6.76KiB	0.0135624	False	False
file_to_blackhole	-6.5KiB	-0.01	5.05%	95.32MiB	3.55MiB	73.48KiB	0.0372024	95.31MiB	3.44MiB	71.56KiB	0.0360995	False	False
splunk_hec_indexer_ack_blackhole	-11.78KiB	-0.05	34.02%	23.75MiB	906.58KiB	18.44KiB	0.0372668	23.74MiB	953.9KiB	19.4KiB	0.039231	False	False
splunk_hec_to_splunk_hec_logs_acks	-17.21KiB	-0.07	54.11%	23.77MiB	770.05KiB	15.68KiB	0.0316305	23.75MiB	843.01KiB	17.15KiB	0.034652	False	False
http_to_http_json	-55.0KiB	-0.23	99.99%	23.85MiB	353.53KiB	7.22KiB	0.0144747	23.79MiB	602.86KiB	12.29KiB	0.0247391	False	False
http_pipelines_blackhole_acks	-4.06KiB	-0.34	86.03%	1.16MiB	110.64KiB	2.25KiB	0.0934111	1.15MiB	76.98KiB	1.57KiB	0.0652198	False	False
fluent_elasticsearch	-303.25KiB	-0.37	100.00%	79.47MiB	54.33KiB	1.1KiB	0.000667479	79.18MiB	3.3MiB	67.78KiB	0.0416328	False	False
http_to_http_noack	-105.61KiB	-0.43	100.00%	23.84MiB	408.05KiB	8.34KiB	0.0167137	23.73MiB	1.15MiB	24.0KiB	0.048497	False	False
datadog_agent_remap_datadog_logs_acks	-363.89KiB	-0.56	99.30%	63.85MiB	3.98MiB	83.09KiB	0.0622421	63.5MiB	5.11MiB	106.33KiB	0.0804261	False	False
datadog_agent_remap_datadog_logs	-451.8KiB	-0.69	100.00%	64.09MiB	2.34MiB	49.11KiB	0.0365143	63.65MiB	4.38MiB	91.28KiB	0.0687925	False	False
syslog_loki	-137.14KiB	-0.87	100.00%	15.35MiB	369.7KiB	7.57KiB	0.023515	15.22MiB	787.63KiB	16.01KiB	0.0505383	False	False
datadog_agent_remap_blackhole_acks	-714.91KiB	-1.1	100.00%	63.58MiB	5.64MiB	117.39KiB	0.0886646	62.88MiB	3.57MiB	74.64KiB	0.0566923	False	False
http_text_to_http_json	-486.98KiB	-1.19	100.00%	39.94MiB	730.51KiB	14.91KiB	0.0178595	39.46MiB	812.17KiB	16.58KiB	0.0200952	False	False
http_pipelines_blackhole	-34.23KiB	-2.06	100.00%	1.62MiB	66.58KiB	1.36KiB	0.0400554	1.59MiB	132.41KiB	2.7KiB	0.0813387	False	False
http_pipelines_no_grok_blackhole	-248.9KiB	-2.15	100.00%	11.31MiB	243.94KiB	4.98KiB	0.0210616	11.07MiB	1.07MiB	22.22KiB	0.0963783	False	False
syslog_regex_logs2metric_ddmetrics	-481.64KiB	-3.64	100.00%	12.92MiB	621.79KiB	12.67KiB	0.0469718	12.45MiB	615.9KiB	12.54KiB	0.0482842	False	False
http_to_http_acks	-972.41KiB	-5.03	100.00%	18.88MiB	6.79MiB	141.95KiB	0.35954	17.93MiB	7.74MiB	161.55KiB	0.431296	True	True
syslog_humio_logs	-963.81KiB	-5.3	100.00%	17.75MiB	139.44KiB	2.85KiB	0.00767218	16.8MiB	146.81KiB	3.01KiB	0.00852981	False	False
syslog_splunk_hec_logs	-985.09KiB	-5.54	100.00%	17.37MiB	894.91KiB	18.22KiB	0.0502984	16.41MiB	773.05KiB	15.78KiB	0.0459966	False	False
syslog_log2metric_splunk_hec_metrics	-1.15MiB	-6.25	100.00%	18.44MiB	717.86KiB	14.62KiB	0.0380136	17.28MiB	729.63KiB	14.86KiB	0.0412137	False	False

Jul 11 '22 23:07 github-actions[bot]

Soak Test Results

Baseline: 2a83dca8fd30b7d8b35a894c3cef0c3930721203 Comparison: 46f4c1082d179b03923536005da51ea3845ccbbd Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_to_http_acks	647.34KiB	3.53	99.63%	17.88MiB	7.94MiB	165.92KiB	0.443678	18.52MiB	7.13MiB	148.78KiB	0.385225	True	True
splunk_hec_route_s3	247.2KiB	1.26	99.99%	19.12MiB	2.22MiB	46.22KiB	0.116071	19.36MiB	2.11MiB	44.05KiB	0.108751	False	False
datadog_agent_remap_datadog_logs_acks	740.71KiB	1.19	100.00%	60.54MiB	3.96MiB	82.73KiB	0.0653599	61.26MiB	4.86MiB	101.09KiB	0.0792545	False	False
syslog_loki	62.31KiB	0.42	99.91%	14.5MiB	459.49KiB	9.41KiB	0.0309446	14.56MiB	801.34KiB	16.29KiB	0.0537415	False	False
http_pipelines_blackhole_acks	1.21KiB	0.1	34.96%	1.14MiB	116.06KiB	2.36KiB	0.0992719	1.14MiB	61.19KiB	1.25KiB	0.0522836	False	False
splunk_hec_to_splunk_hec_logs_noack	6.08KiB	0.02	44.66%	23.83MiB	381.92KiB	7.8KiB	0.0156467	23.84MiB	326.18KiB	6.66KiB	0.0133598	False	False
syslog_regex_logs2metric_ddmetrics	1.77KiB	0.01	7.05%	12.98MiB	685.5KiB	13.96KiB	0.051547	12.99MiB	699.7KiB	14.26KiB	0.0526073	False	False
file_to_blackhole	-51.52KiB	-0.05	43.55%	95.36MiB	2.82MiB	58.39KiB	0.0295307	95.31MiB	3.26MiB	67.73KiB	0.0341681	False	False
syslog_humio_logs	-11.68KiB	-0.06	86.90%	17.7MiB	263.15KiB	5.37KiB	0.0145178	17.69MiB	271.72KiB	5.56KiB	0.0149998	False	False
splunk_hec_to_splunk_hec_logs_acks	-16.46KiB	-0.07	52.26%	23.77MiB	767.51KiB	15.62KiB	0.0315281	23.75MiB	840.51KiB	17.1KiB	0.0345503	False	False
splunk_hec_indexer_ack_blackhole	-16.18KiB	-0.07	46.76%	23.76MiB	856.55KiB	17.43KiB	0.0351968	23.74MiB	942.32KiB	19.17KiB	0.038747	False	False
http_to_http_json	-29.87KiB	-0.12	98.44%	23.85MiB	338.24KiB	6.91KiB	0.013849	23.82MiB	501.05KiB	10.24KiB	0.0205403	False	False
syslog_splunk_hec_logs	-34.41KiB	-0.2	85.70%	17.01MiB	824.69KiB	16.79KiB	0.0473315	16.98MiB	805.65KiB	16.42KiB	0.0463302	False	False
syslog_log2metric_splunk_hec_metrics	-39.48KiB	-0.2	94.03%	19.2MiB	602.82KiB	12.29KiB	0.0306605	19.16MiB	834.17KiB	16.98KiB	0.0425131	False	False
fluent_elasticsearch	-190.35KiB	-0.23	100.00%	79.47MiB	54.24KiB	1.1KiB	0.000666316	79.29MiB	1.73MiB	35.58KiB	0.0217916	False	False
http_text_to_http_json	-126.97KiB	-0.32	100.00%	38.98MiB	787.47KiB	16.07KiB	0.019722	38.86MiB	820.04KiB	16.75KiB	0.0206031	False	False
datadog_agent_remap_blackhole	-231.43KiB	-0.35	96.31%	64.67MiB	4.5MiB	93.69KiB	0.0695678	64.45MiB	2.84MiB	59.26KiB	0.0440492	False	False
http_to_http_noack	-87.68KiB	-0.36	99.99%	23.85MiB	239.07KiB	4.89KiB	0.00978862	23.76MiB	1.02MiB	21.21KiB	0.0427907	False	False
datadog_agent_remap_datadog_logs	-338.11KiB	-0.53	99.97%	61.97MiB	1.23MiB	25.87KiB	0.0199014	61.64MiB	4.3MiB	89.63KiB	0.0698064	False	False
socket_to_socket_blackhole	-94.5KiB	-0.68	100.00%	13.55MiB	272.54KiB	5.56KiB	0.0196394	13.46MiB	302.83KiB	6.18KiB	0.0219718	False	False
syslog_log2metric_humio_metrics	-123.17KiB	-0.91	100.00%	13.27MiB	287.2KiB	5.86KiB	0.0211343	13.15MiB	527.82KiB	10.75KiB	0.0391962	False	False
http_pipelines_blackhole	-17.45KiB	-1.05	100.00%	1.62MiB	39.09KiB	818.75B	0.0235884	1.6MiB	110.36KiB	2.25KiB	0.0673077	False	False
http_pipelines_no_grok_blackhole	-244.19KiB	-2.09	100.00%	11.39MiB	49.32KiB	1.01KiB	0.00422725	11.15MiB	1.14MiB	23.66KiB	0.101909	False	False
datadog_agent_remap_blackhole_acks	-1.81MiB	-2.76	100.00%	65.47MiB	4.75MiB	98.81KiB	0.0724907	63.66MiB	4.33MiB	90.63KiB	0.0679704	False	False

Jul 12 '22 16:07 github-actions[bot]

Hi everyone! Thanks for addressing this bug -- Is this fix still blocked? I am testing a new vector deployment on a high volume kafka topic. and believe I have hit this a few times during testing; would be happy to help test this if needed!

Aug 29 '22 20:08 jches

We have merged support for a relevant version of rdkafka (#10302), but there still appear to be some linking issues on the less common platforms. It probably will need to be re-merged after #14135 lands, so there is a little work to do yet.

Aug 29 '22 20:08 bruceg

@neuronull the build failures here look a lot like the OpenSSL build failures we resolved before when upgrading rdkafka. Does anything jump out to you?

Sep 21 '22 19:09 jszwedko

@neuronull the build failures here look a lot like the OpenSSL build failures we resolved before when upgrading rdkafka. Does anything jump out to you?

It does look very similar. I tried to dig up the exact build failures we saw before, but I'm not finding them. But I can say, it was during the linking stage, and it was the ENGINE feature set that the linking failed on.

At the moment not able to think of why this might be coming up again. Just glancing at the changes here it looks like the rust-rdkafka version is changed. I wonder if that might impact anything :thought_balloon:

Sep 22 '22 14:09 neuronull

This is an old PR that might be fixed by simply merging in current master branch. However, the sources for the kafka source have diverted significantly since the last merge, so that merge will be non-trivial.

Sep 22 '22 21:09 bruceg

Approach still looks good to me. It's unfortunate that we can't test it.

I've been watching this one since we saw this bug manifest pretty regularly when we first started testing vector. We've been running a patched version of Vector where we cherry-picked this branch over, and I can at least confirm that we haven't seen the backwards offset reset since applying this patch (although we also took steps to reduce the number of group rebalances that were happening).

The approach seems good, though I agree with one of the discussions above suggesting a follow-up to correct the problem of flushing acknowledgements for partitions that are not revoked during a rebalance.

Oct 12 '22 14:10 jches

Thanks for the note @jches, I'll be sure to add a follow-up issue once this is merged.

Oct 14 '22 17:10 bruceg

vector
vector copied to clipboard

fix(kafka source): Flush commits on rebalance events

Deploy Preview for vector-project canceled.

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

vector vector copied to clipboard

fix(kafka source): Flush commits on rebalance events

✅ Deploy Preview for vector-project canceled.

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

vector
vector copied to clipboard

Deploy Preview for vector-project canceled.