enhancement(observability): Add initial registered internal event implementation

[bruceg]

This adds the new RegisteredBytesSent event and puts it into use in all sinks except for the new-style stream sinks. That change is blocked by a complication in how the driver handles emitting the BytesSent event centrally.

Ref: #13691

[netlify[bot]]

✅ Deploy Preview for vector-project ready!

Name	Link
🔨 Latest commit	42196db431b5e1a97ce1b1e5d4fd5c82f5336a3c
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62f41238bbcf0b000992d7e6
😎 Deploy Preview	https://deploy-preview-13905--vector-project.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions[bot]]

Soak Test Results

Baseline: afcee9be8840174292d47a9e06d19d059f77e246 Comparison: b33cef8e107f27da9ae293ac0250ed6f29b8fb19 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_humio_logs	152.7KiB	0.93	100.00%	16.09MiB	386.75KiB	7.9KiB	0	0.0234708	16.24MiB	401.49KiB	8.22KiB	0	0.0241417	False	False
syslog_regex_logs2metric_ddmetrics	103.39KiB	0.81	100.00%	12.41MiB	581.13KiB	11.84KiB	0	0.0457127	12.51MiB	556.92KiB	11.35KiB	0	0.043455	False	False
syslog_log2metric_humio_metrics	63.86KiB	0.5	99.14%	12.59MiB	749.76KiB	15.3KiB	0	0.0581415	12.65MiB	926.78KiB	18.86KiB	0	0.0715139	False	False
splunk_hec_to_splunk_hec_logs_noack	28.33KiB	0.12	96.94%	23.81MiB	548.33KiB	11.19KiB	0	0.0224838	23.84MiB	334.03KiB	6.82KiB	0	0.0136804	False	False
datadog_agent_remap_blackhole_acks	62.5KiB	0.09	43.51%	64.76MiB	4.5MiB	93.74KiB	0	0.0695109	64.82MiB	2.62MiB	54.81KiB	0	0.0403918	False	False
http_pipelines_blackhole_acks	99.78B	0.01	2.96%	1.17MiB	108.8KiB	2.21KiB	0	0.0907935	1.17MiB	69.34KiB	1.41KiB	0	0.0578594	False	False
enterprise_http_to_http	-804.19B	-0	8.76%	23.85MiB	248.69KiB	5.08KiB	0	0.0101823	23.85MiB	245.32KiB	5.02KiB	0	0.0100448	False	False
splunk_hec_indexer_ack_blackhole	-1.28KiB	-0.01	3.90%	23.75MiB	908.34KiB	18.48KiB	0	0.0373456	23.75MiB	912.6KiB	18.56KiB	0	0.0375226	False	False
splunk_hec_to_splunk_hec_logs_acks	-14.96KiB	-0.06	48.80%	23.77MiB	757.76KiB	15.42KiB	0	0.0311259	23.75MiB	825.94KiB	16.8KiB	0	0.0339476	False	False
file_to_blackhole	-64.69KiB	-0.07	54.37%	95.36MiB	2.73MiB	56.55KiB	0	0.0285972	95.3MiB	3.17MiB	65.88KiB	0	0.0332157	False	False
splunk_hec_route_s3	-16.61KiB	-0.09	18.62%	18.08MiB	2.41MiB	50.16KiB	0	0.13322	18.07MiB	2.37MiB	49.56KiB	0	0.131269	False	False
http_to_http_json	-35.59KiB	-0.15	99.49%	23.85MiB	346.61KiB	7.08KiB	0	0.0141905	23.81MiB	515.86KiB	10.54KiB	0	0.0211503	False	False
datadog_agent_remap_blackhole	-96.98KiB	-0.15	61.93%	62.76MiB	4.2MiB	87.58KiB	0	0.0669578	62.66MiB	3.24MiB	67.58KiB	0	0.0516856	False	False
syslog_log2metric_splunk_hec_metrics	-30.16KiB	-0.17	79.16%	17.39MiB	714.96KiB	14.58KiB	0	0.0401487	17.36MiB	935.07KiB	19.04KiB	0	0.052598	False	False
fluent_elasticsearch	-151.34KiB	-0.19	100.00%	79.47MiB	53.31KiB	1.08KiB	0	0.000654929	79.32MiB	1.42MiB	29.29KiB	0	0.0179235	False	False
http_to_http_noack	-103.5KiB	-0.42	100.00%	23.84MiB	413.36KiB	8.46KiB	0	0.016931	23.74MiB	1.14MiB	23.68KiB	0	0.0478396	False	False
syslog_splunk_hec_logs	-80.12KiB	-0.48	99.59%	16.16MiB	952.97KiB	19.39KiB	0	0.0575726	16.08MiB	982.56KiB	20.01KiB	0	0.0596494	False	False
http_to_http_acks	-119.47KiB	-0.67	39.39%	17.32MiB	8.11MiB	169.49KiB	0	0.468191	17.2MiB	7.57MiB	157.98KiB	0	0.44	True	True
http_pipelines_blackhole	-15.21KiB	-0.93	100.00%	1.6MiB	54.88KiB	1.12KiB	0	0.0335741	1.58MiB	104.51KiB	2.13KiB	0	0.0645363	False	False
syslog_loki	-152.68KiB	-1.05	100.00%	14.18MiB	377.36KiB	7.72KiB	0	0.0259779	14.03MiB	706.39KiB	14.36KiB	0	0.0491449	False	False
http_pipelines_no_grok_blackhole	-196.44KiB	-1.75	100.00%	10.97MiB	225.55KiB	4.61KiB	0	0.0200811	10.77MiB	1.04MiB	21.67KiB	0	0.0965563	False	False
datadog_agent_remap_datadog_logs	-1.11MiB	-1.79	100.00%	61.98MiB	409.32KiB	8.38KiB	0	0.00644763	60.87MiB	3.95MiB	82.25KiB	0	0.0648538	False	False
datadog_agent_remap_datadog_logs_acks	-1.68MiB	-2.68	100.00%	62.77MiB	2.7MiB	56.43KiB	0	0.042952	61.09MiB	4.43MiB	92.16KiB	0	0.0724616	False	False
http_text_to_http_json	-1.82MiB	-4.61	100.00%	39.38MiB	769.25KiB	15.7KiB	0	0.0190742	37.56MiB	841.11KiB	17.17KiB	0	0.0218649	False	False
socket_to_socket_blackhole	-1.22MiB	-5.15	100.00%	23.67MiB	211.86KiB	4.32KiB	0	0.00873856	22.45MiB	166.75KiB	3.4KiB	0	0.00725112	False	False

[github-actions[bot]]

Soak Test Results

Baseline: a47dc196134e1f19b4c712a38541437879b33a08 Comparison: 03ed2ac97d4679fd1df61b17cc06c14d5292263e Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_text_to_http_json	1.2MiB	3.29	100.00%	36.49MiB	1.24MiB	25.9KiB	0	0.0339434	37.69MiB	1.41MiB	29.48KiB	0	0.0374107	False	False
socket_to_socket_blackhole	601.7KiB	2.76	100.00%	21.31MiB	1.01MiB	21.2KiB	0	0.0475677	21.9MiB	1.14MiB	23.78KiB	0	0.0519335	False	False
splunk_hec_route_s3	389.33KiB	2.16	100.00%	17.59MiB	2.41MiB	50.07KiB	0	0.136749	17.97MiB	2.34MiB	48.84KiB	0	0.130045	False	False
http_pipelines_blackhole	30.42KiB	2.01	100.00%	1.48MiB	108.51KiB	2.22KiB	0	0.0715471	1.51MiB	152.16KiB	3.1KiB	0	0.0983536	False	False
syslog_splunk_hec_logs	234.12KiB	1.39	100.00%	16.45MiB	759.32KiB	15.44KiB	0	0.0450608	16.68MiB	573.38KiB	11.71KiB	0	0.03356	False	False
datadog_agent_remap_datadog_logs_acks	710.02KiB	1.14	100.00%	60.68MiB	3.84MiB	80.15KiB	0	0.0633119	61.37MiB	4.44MiB	92.51KiB	0	0.0723973	False	False
syslog_regex_logs2metric_ddmetrics	141.49KiB	1.11	100.00%	12.45MiB	568.19KiB	11.58KiB	0	0.0445733	12.58MiB	509.19KiB	10.38KiB	0	0.0395067	False	False
datadog_agent_remap_blackhole_acks	630.18KiB	1	100.00%	61.24MiB	5.04MiB	105.09KiB	0	0.0823606	61.85MiB	3.98MiB	83.25KiB	0	0.0643381	False	False
http_to_http_acks	160.78KiB	0.9	51.07%	17.36MiB	7.7MiB	160.9KiB	0	0.443201	17.52MiB	8.05MiB	167.86KiB	0	0.459296	True	True
syslog_loki	125.85KiB	0.89	100.00%	13.84MiB	598.4KiB	12.26KiB	0	0.0422186	13.96MiB	893.27KiB	18.16KiB	0	0.0624676	False	False
syslog_log2metric_splunk_hec_metrics	139.83KiB	0.77	100.00%	17.8MiB	567.54KiB	11.57KiB	0	0.0311251	17.94MiB	773.96KiB	15.75KiB	0	0.0421227	False	False
syslog_humio_logs	126.07KiB	0.76	100.00%	16.28MiB	286.41KiB	5.85KiB	0	0.0171767	16.4MiB	322.98KiB	6.61KiB	0	0.0192244	False	False
datadog_agent_remap_datadog_logs	408.91KiB	0.64	100.00%	62.77MiB	327.69KiB	6.71KiB	0	0.00509736	63.17MiB	4.36MiB	90.82KiB	0	0.0690413	False	False
http_pipelines_blackhole_acks	6.62KiB	0.55	98.75%	1.17MiB	105.06KiB	2.14KiB	0	0.0875851	1.18MiB	76.71KiB	1.56KiB	0	0.0635957	False	False
syslog_log2metric_humio_metrics	65.38KiB	0.52	99.99%	12.33MiB	493.64KiB	10.08KiB	0	0.0390821	12.4MiB	646.65KiB	13.17KiB	0	0.050933	False	False
datadog_agent_remap_blackhole	206.08KiB	0.32	92.63%	63.26MiB	4.33MiB	90.1KiB	0	0.0683608	63.46MiB	3.44MiB	71.82KiB	0	0.0542499	False	False
splunk_hec_to_splunk_hec_logs_acks	18.22KiB	0.07	54.40%	23.74MiB	883.71KiB	17.97KiB	0	0.0363391	23.76MiB	814.02KiB	16.56KiB	0	0.0334483	False	False
splunk_hec_to_splunk_hec_logs_noack	9.19KiB	0.04	59.31%	23.83MiB	427.43KiB	8.72KiB	0	0.0175135	23.84MiB	334.12KiB	6.82KiB	0	0.0136852	False	False
splunk_hec_indexer_ack_blackhole	7.85KiB	0.03	26.47%	23.75MiB	812.46KiB	16.52KiB	0	0.0333971	23.76MiB	802.63KiB	16.33KiB	0	0.0329824	False	False
enterprise_http_to_http	-1.7KiB	-0.01	17.89%	23.85MiB	262.28KiB	5.35KiB	0	0.0107382	23.85MiB	259.17KiB	5.3KiB	0	0.0106116	False	False
http_to_http_json	-29.33KiB	-0.12	98.17%	23.85MiB	344.06KiB	7.02KiB	0	0.0140872	23.82MiB	501.85KiB	10.26KiB	0	0.0205724	False	False
file_to_blackhole	-151.03KiB	-0.15	79.40%	95.32MiB	3.47MiB	71.83KiB	0	0.0363973	95.17MiB	4.6MiB	95.39KiB	0	0.0483068	False	False
http_pipelines_no_grok_blackhole	-20.86KiB	-0.19	59.49%	10.61MiB	421.83KiB	8.61KiB	0	0.0388041	10.59MiB	1.13MiB	23.53KiB	0	0.106648	False	False
fluent_elasticsearch	-283.03KiB	-0.35	100.00%	79.47MiB	55.14KiB	1.11KiB	0	0.000677412	79.2MiB	2.63MiB	54.11KiB	0	0.0332428	False	False
http_to_http_noack	-96.15KiB	-0.39	100.00%	23.85MiB	246.68KiB	5.05KiB	0	0.0101002	23.75MiB	1.06MiB	22.18KiB	0	0.0447706	False	False

[github-actions[bot]]

Soak Test Results

Baseline: 92a5082f1d02f0c45ff74963b06bc64c87fe7a95 Comparison: 42196db431b5e1a97ce1b1e5d4fd5c82f5336a3c Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_text_to_http_json	297.97KiB	0.75	100.00%	38.6MiB	1.13MiB	23.58KiB	0	0.0292262	38.89MiB	1.21MiB	25.26KiB	0	0.0310741	False	False
splunk_hec_to_splunk_hec_logs_noack	27.26KiB	0.11	96.59%	23.81MiB	530.51KiB	10.82KiB	0	0.0217525	23.84MiB	340.62KiB	6.95KiB	0	0.0139506	False	False
splunk_hec_indexer_ack_blackhole	25.48KiB	0.1	64.45%	23.73MiB	1008.89KiB	20.51KiB	0	0.0415151	23.75MiB	905.49KiB	18.42KiB	0	0.0372211	False	False
enterprise_http_to_http	-4.54KiB	-0.02	45.13%	23.85MiB	262.73KiB	5.36KiB	0	0.0107563	23.84MiB	261.4KiB	5.35KiB	0	0.0107036	False	False
splunk_hec_to_splunk_hec_logs_acks	-10.28KiB	-0.04	32.18%	23.75MiB	835.51KiB	16.99KiB	0	0.0343448	23.74MiB	886.83KiB	18.03KiB	0	0.0364699	False	False
http_to_http_json	-33.12KiB	-0.14	99.02%	23.85MiB	352.52KiB	7.2KiB	0	0.0144335	23.81MiB	519.15KiB	10.61KiB	0	0.0212845	False	False
splunk_hec_route_s3	-36.05KiB	-0.19	42.37%	18.96MiB	2.24MiB	46.69KiB	0	0.11825	18.92MiB	2.13MiB	44.53KiB	0	0.112407	False	False
fluent_elasticsearch	-214.33KiB	-0.26	100.00%	79.47MiB	52.85KiB	1.07KiB	0	0.000649279	79.26MiB	2.08MiB	42.67KiB	0	0.0261752	False	False
datadog_agent_remap_blackhole_acks	-178.41KiB	-0.27	87.11%	63.4MiB	4.85MiB	100.89KiB	0	0.0763986	63.23MiB	2.88MiB	60.19KiB	0	0.0454779	False	False
file_to_blackhole	-307.18KiB	-0.31	97.64%	95.33MiB	3.37MiB	69.78KiB	0	0.0353364	95.03MiB	5.61MiB	116.31KiB	0	0.0589891	False	False
syslog_humio_logs	-53.4KiB	-0.32	100.00%	16.25MiB	289.09KiB	5.9KiB	0	0.0173731	16.19MiB	305.68KiB	6.26KiB	0	0.0184289	False	False
http_to_http_noack	-89.7KiB	-0.37	100.00%	23.84MiB	261.53KiB	5.35KiB	0	0.0107095	23.76MiB	1.03MiB	21.39KiB	0	0.0431741	False	False
syslog_log2metric_splunk_hec_metrics	-90.61KiB	-0.52	99.99%	16.99MiB	741.29KiB	15.11KiB	0	0.0426014	16.9MiB	851.98KiB	17.34KiB	0	0.0492193	False	False
syslog_splunk_hec_logs	-86.91KiB	-0.52	100.00%	16.43MiB	756.21KiB	15.39KiB	0	0.0449513	16.34MiB	607.71KiB	12.39KiB	0	0.0363114	False	False
http_pipelines_blackhole	-8.82KiB	-0.53	99.99%	1.63MiB	12.24KiB	256.07B	0	0.00732699	1.62MiB	105.96KiB	2.16KiB	0	0.0637746	False	False
syslog_regex_logs2metric_ddmetrics	-66.25KiB	-0.54	99.99%	12.09MiB	602.24KiB	12.27KiB	0	0.0486436	12.02MiB	549.77KiB	11.2KiB	0	0.0446445	False	False
datadog_agent_remap_datadog_logs	-533.13KiB	-0.83	100.00%	62.48MiB	1.14MiB	23.94KiB	0	0.0182814	61.96MiB	4.17MiB	86.73KiB	0	0.0672034	False	False
http_to_http_acks	-168.5KiB	-0.94	52.37%	17.54MiB	8.16MiB	170.55KiB	0	0.464953	17.38MiB	7.87MiB	163.93KiB	0	0.452632	True	True
socket_to_socket_blackhole	-249.7KiB	-1.08	100.00%	22.59MiB	1.09MiB	22.74KiB	0	0.0481404	22.35MiB	1.05MiB	21.92KiB	0	0.0469188	False	False
datadog_agent_remap_datadog_logs_acks	-743.02KiB	-1.17	100.00%	62.02MiB	3.7MiB	77.39KiB	0	0.0596331	61.3MiB	5.14MiB	106.93KiB	0	0.0837844	False	False
http_pipelines_no_grok_blackhole	-131.82KiB	-1.2	100.00%	10.74MiB	488.83KiB	9.98KiB	0	0.0444419	10.61MiB	1.13MiB	23.5KiB	0	0.106355	False	False
http_pipelines_blackhole_acks	-17.96KiB	-1.57	100.00%	1.12MiB	127.18KiB	2.59KiB	0	0.110921	1.1MiB	101.79KiB	2.07KiB	0	0.090192	False	False
syslog_log2metric_humio_metrics	-240.04KiB	-1.8	100.00%	13.0MiB	249.44KiB	5.09KiB	0	0.0187399	12.76MiB	537.89KiB	10.95KiB	0	0.0411527	False	False
syslog_loki	-288.22KiB	-1.91	100.00%	14.73MiB	495.34KiB	10.14KiB	0	0.0328245	14.45MiB	728.27KiB	14.8KiB	0	0.0492005	False	False
datadog_agent_remap_blackhole	-2.17MiB	-3.5	100.00%	62.18MiB	5.46MiB	113.81KiB	0	0.0877824	60.01MiB	5.52MiB	115.12KiB	0	0.0919494	False	False