feat(kafka): synchronous consumer offset commit on shutdown

[aholmberg]

trafficstars

Make one final attempt to commit stored offsets on kafka source shutdown.

[bits-bot]

All committers have signed the CLA.

[netlify[bot]]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	41929c27859d730910443250a86b4b5bc2053200
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63bd8ec1ebf9960008ed345b

[netlify[bot]]

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	41929c27859d730910443250a86b4b5bc2053200
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63bd8ec1c1734b00083e43d8

[bruceg]

Please run cargo fmt and submit the changes.

[github-actions[bot]]

Regression Test Results

Run ID: c4aaa3ef-a741-417c-9fba-8a6960fc08fc
Baseline: cbe52f0c5bd7ed4161f7b8fa36ac88ec27b2bffa
Comparison: 41929c27859d730910443250a86b4b5bc2053200
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment	Δ mean	Δ mean %	confidence
syslog_loki	-643.05KiB/CPU-s	-7.0	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_regex_logs2metric_ddmetrics	61.39KiB/CPU-s	1.67	100.00%	3.6MiB/CPU-s	500.1KiB/CPU-s	6.46KiB/CPU-s	0.0	0.135833	3.66MiB/CPU-s	496.75KiB/CPU-s	6.41KiB/CPU-s	0.0	0.132708	True	False
otlp_http_to_blackhole	19.88KiB/CPU-s	1.25	100.00%	1.55MiB/CPU-s	102.46KiB/CPU-s	1.32KiB/CPU-s	0.0	0.064532	1.57MiB/CPU-s	104.88KiB/CPU-s	1.35KiB/CPU-s	0.0	0.065236	False	False
datadog_agent_remap_blackhole_acks	372.22KiB/CPU-s	1.18	100.00%	30.73MiB/CPU-s	1.63MiB/CPU-s	21.61KiB/CPU-s	0.0	0.053191	31.09MiB/CPU-s	1.39MiB/CPU-s	18.42KiB/CPU-s	0.0	0.044801	False	False
splunk_hec_route_s3	93.22KiB/CPU-s	0.77	100.00%	11.81MiB/CPU-s	531.07KiB/CPU-s	6.85KiB/CPU-s	0.0	0.043921	11.9MiB/CPU-s	517.54KiB/CPU-s	6.68KiB/CPU-s	0.0	0.042475	False	False
enterprise_http_to_http	6.16KiB/CPU-s	0.04	78.42%	13.61MiB/CPU-s	310.82KiB/CPU-s	4.01KiB/CPU-s	0.0	0.022294	13.62MiB/CPU-s	228.05KiB/CPU-s	2.94KiB/CPU-s	0.0	0.01635	False	False
datadog_agent_remap_datadog_logs_acks	9.27KiB/CPU-s	0.03	29.63%	33.91MiB/CPU-s	1.41MiB/CPU-s	18.65KiB/CPU-s	0.0	0.041623	33.92MiB/CPU-s	1.19MiB/CPU-s	15.7KiB/CPU-s	0.0	0.035029	False	False
splunk_hec_to_splunk_hec_logs_noack	3.34KiB/CPU-s	0.02	63.01%	13.62MiB/CPU-s	216.27KiB/CPU-s	2.79KiB/CPU-s	0.0	0.015503	13.63MiB/CPU-s	190.73KiB/CPU-s	2.46KiB/CPU-s	0.0	0.013669	False	False
splunk_hec_to_splunk_hec_logs_acks	1.15KiB/CPU-s	0.01	14.58%	13.61MiB/CPU-s	344.82KiB/CPU-s	4.45KiB/CPU-s	0.0	0.024732	13.62MiB/CPU-s	339.14KiB/CPU-s	4.38KiB/CPU-s	0.0	0.024322	False	False
fluent_elasticsearch	224.2B/CPU-s	0.0	31.59%	45.41MiB/CPU-s	29.7KiB/CPU-s	388.38B/CPU-s	0.0	0.000639	45.41MiB/CPU-s	29.89KiB/CPU-s	390.96B/CPU-s	0.0	0.000643	False	False
http_to_http_json	304.44B/CPU-s	0.0	6.12%	13.62MiB/CPU-s	214.0KiB/CPU-s	2.76KiB/CPU-s	0.0	0.01534	13.62MiB/CPU-s	210.19KiB/CPU-s	2.71KiB/CPU-s	0.0	0.015066	False	False
socket_to_socket_blackhole	-26.62B/CPU-s	-0.0	0.37%	13.68MiB/CPU-s	191.77KiB/CPU-s	2.48KiB/CPU-s	0.0	0.013693	13.68MiB/CPU-s	384.98KiB/CPU-s	4.97KiB/CPU-s	0.0	0.02749	False	False
file_to_blackhole	-10.77KiB/CPU-s	-0.02	37.96%	54.5MiB/CPU-s	1.15MiB/CPU-s	15.25KiB/CPU-s	0.0	0.021178	54.49MiB/CPU-s	1.17MiB/CPU-s	15.49KiB/CPU-s	0.0	0.021535	False	False
splunk_hec_indexer_ack_blackhole	-2.47KiB/CPU-s	-0.02	40.84%	13.62MiB/CPU-s	247.28KiB/CPU-s	3.19KiB/CPU-s	0.0	0.01773	13.62MiB/CPU-s	257.06KiB/CPU-s	3.32KiB/CPU-s	0.0	0.018435	False	False
http_to_http_noack	-4.39KiB/CPU-s	-0.03	56.32%	13.62MiB/CPU-s	282.35KiB/CPU-s	3.64KiB/CPU-s	0.0	0.020249	13.61MiB/CPU-s	333.97KiB/CPU-s	4.31KiB/CPU-s	0.0	0.023959	False	False
datadog_agent_remap_datadog_logs	-23.66KiB/CPU-s	-0.07	64.73%	33.94MiB/CPU-s	1.35MiB/CPU-s	17.85KiB/CPU-s	0.0	0.039813	33.92MiB/CPU-s	1.37MiB/CPU-s	18.16KiB/CPU-s	0.0	0.040532	False	False
datadog_agent_remap_blackhole	-27.54KiB/CPU-s	-0.09	77.61%	31.11MiB/CPU-s	978.41KiB/CPU-s	12.64KiB/CPU-s	0.0	0.030708	31.09MiB/CPU-s	1.42MiB/CPU-s	18.8KiB/CPU-s	0.0	0.045749	False	False
http_text_to_http_json	-200.05KiB/CPU-s	-0.77	100.00%	25.37MiB/CPU-s	673.73KiB/CPU-s	8.69KiB/CPU-s	0.0	0.025935	25.17MiB/CPU-s	568.82KiB/CPU-s	7.34KiB/CPU-s	0.0	0.022067	False	False
http_to_http_acks	-45.3KiB/CPU-s	-0.83	62.45%	5.3MiB/CPU-s	2.72MiB/CPU-s	35.97KiB/CPU-s	0.0	0.513691	5.26MiB/CPU-s	2.75MiB/CPU-s	36.31KiB/CPU-s	0.0	0.522804	True	False
syslog_splunk_hec_logs	-90.63KiB/CPU-s	-0.96	100.00%	9.27MiB/CPU-s	68.24KiB/CPU-s	902.17B/CPU-s	0.0	0.007193	9.18MiB/CPU-s	152.88KiB/CPU-s	1.97KiB/CPU-s	0.0	0.016268	False	False
syslog_humio_logs	-122.59KiB/CPU-s	-1.31	100.00%	9.11MiB/CPU-s	478.97KiB/CPU-s	6.18KiB/CPU-s	0.0	0.051342	8.99MiB/CPU-s	472.19KiB/CPU-s	6.1KiB/CPU-s	0.0	0.05129	False	False
otlp_grpc_to_blackhole	-14.64KiB/CPU-s	-1.36	100.00%	1.05MiB/CPU-s	39.59KiB/CPU-s	523.28B/CPU-s	0.0	0.036662	1.04MiB/CPU-s	57.57KiB/CPU-s	760.67B/CPU-s	0.0	0.054045	False	False
syslog_log2metric_humio_metrics	-113.8KiB/CPU-s	-1.85	100.00%	6.0MiB/CPU-s	360.56KiB/CPU-s	4.65KiB/CPU-s	0.0	0.058709	5.89MiB/CPU-s	394.42KiB/CPU-s	5.09KiB/CPU-s	0.0	0.065434	False	False
syslog_log2metric_splunk_hec_metrics	-195.65KiB/CPU-s	-2.03	100.00%	9.42MiB/CPU-s	415.74KiB/CPU-s	5.37KiB/CPU-s	0.0	0.043094	9.23MiB/CPU-s	395.64KiB/CPU-s	5.11KiB/CPU-s	0.0	0.041859	False	False
syslog_loki	-643.05KiB/CPU-s	-7.0	100.00%	8.97MiB/CPU-s	256.96KiB/CPU-s	3.32KiB/CPU-s	0.0	0.027965	8.34MiB/CPU-s	511.41KiB/CPU-s	6.6KiB/CPU-s	0.0	0.059843	False	False

[aholmberg]

@bruceg I don't know if it notified, but I did push with the format fixed.