VRL: Refactor type definitions to handle side-effects in blocks

[fuchsnj]

closes https://github.com/vectordotdev/vector/issues/13527

There were many unrelated issues opened while making these changes. I attempted to only fix the original issue to keep this change as small as possible.

Previously, the Expression::type_def(..) function was passed in the final state of the environment. The result of expressions could be calculated correctly from this, but it prevented fallibility or termination checking from working correctly. Now, the initial state is passed in, and a new function (Expression::type_info(..)) was added, so that accurate type information, including side-effects such as assignments, can be propagated correctly.

A new FunctionExpression trait was added to help reduce the number of changes necessary. Most functions do not modify state, so this new trait tries to enforce that behavior and simplifies the implementation for functions.

Function arguments are in an inconsistent state. The type definitions are calculated as if they are resolved before the function is called, but they are not actually resolved today until after the function starts running. The type state passed into a function being compiled (or when type_def is called for a function) is the state before the function ran, before all arguments have resolved, and after the closure has ran. This is incorrect, but necessary to ensure that calling type_def (or similar functions) on function arguments still returns the correct result. This will be cleaned up / fixed for function arguments and function closures in the future.

ExternalEnv was refactored to only include type information from the event and metadata. The additional information (read only paths, and "external context") was moved to a separate CompileConfig struct. Since expressions are now expected to be able to calculate the resulting state, it didn't make sense to include the other information here.

LocalEnv and ExternalEnv were combined into a single TypeState that can be passed around as a single type.

[netlify[bot]]

✅ Deploy Preview for vector-project ready!

Name	Link
🔨 Latest commit	c065926839838c9542521afbf320ebaa403be122
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62f65945ff6a920008fb0082
😎 Deploy Preview	https://deploy-preview-13833--vector-project.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions[bot]]

Soak Test Results

Baseline: f64bceffe36cf987b23bd541bebc5ef8498d8171 Comparison: d2b41421939a7599c4a05ba2326cf7adef2f9e3d Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	42.83KiB	3.67	100.00%	1.14MiB	118.96KiB	2.42KiB	0	0.101911	1.18MiB	95.18KiB	1.94KiB	0	0.0786513	False	False
http_pipelines_blackhole	58.53KiB	3.62	100.00%	1.58MiB	78.73KiB	1.61KiB	0	0.0487498	1.63MiB	134.64KiB	2.74KiB	0	0.0804564	False	False
syslog_regex_logs2metric_ddmetrics	21.52KiB	0.17	78.08%	12.54MiB	661.62KiB	13.47KiB	0	0.0515014	12.56MiB	549.28KiB	11.2KiB	0	0.0426848	False	False
datadog_agent_remap_blackhole	96.87KiB	0.15	62.66%	61.81MiB	4.23MiB	88.22KiB	0	0.0684238	61.91MiB	3.05MiB	63.69KiB	0	0.0492732	False	False
splunk_hec_to_splunk_hec_logs_noack	15.95KiB	0.07	83.14%	23.82MiB	459.05KiB	9.37KiB	0	0.0188148	23.84MiB	333.52KiB	6.81KiB	0	0.0136609	False	False
splunk_hec_to_splunk_hec_logs_acks	5.16KiB	0.02	16.80%	23.75MiB	841.8KiB	17.12KiB	0	0.0346017	23.76MiB	847.88KiB	17.25KiB	0	0.034844	False	False
enterprise_http_to_http	-807.32B	-0	8.73%	23.85MiB	246.25KiB	5.03KiB	0	0.0100826	23.85MiB	251.52KiB	5.14KiB	0	0.0102985	False	False
file_to_blackhole	-54.66KiB	-0.06	38.68%	95.33MiB	3.51MiB	72.8KiB	0	0.0368311	95.28MiB	3.85MiB	79.92KiB	0	0.0403649	False	False
splunk_hec_indexer_ack_blackhole	-14.07KiB	-0.06	40.26%	23.75MiB	891.04KiB	18.13KiB	0	0.0366252	23.74MiB	960.55KiB	19.53KiB	0	0.0395053	False	False
http_to_http_json	-44.71KiB	-0.18	99.90%	23.85MiB	338.81KiB	6.92KiB	0	0.0138723	23.8MiB	569.29KiB	11.62KiB	0	0.0233517	False	False
fluent_elasticsearch	-180.61KiB	-0.22	100.00%	79.47MiB	53.2KiB	1.07KiB	0	0.000653642	79.3MiB	1.55MiB	31.98KiB	0	0.0196039	False	False
syslog_humio_logs	-57.29KiB	-0.33	100.00%	16.92MiB	321.46KiB	6.56KiB	0	0.018545	16.87MiB	257.48KiB	5.27KiB	0	0.014903	False	False
http_to_http_noack	-92.11KiB	-0.38	99.99%	23.84MiB	404.96KiB	8.29KiB	0	0.0165868	23.75MiB	1.07MiB	22.38KiB	0	0.0451837	False	False
syslog_log2metric_splunk_hec_metrics	-94.07KiB	-0.49	100.00%	18.63MiB	717.87KiB	14.62KiB	0	0.0376127	18.54MiB	793.77KiB	16.16KiB	0	0.0417953	False	False
http_pipelines_no_grok_blackhole	-64.96KiB	-0.57	99.77%	11.16MiB	118.29KiB	2.41KiB	0	0.0103502	11.1MiB	1.02MiB	21.17KiB	0	0.0915795	False	False
syslog_splunk_hec_logs	-130.95KiB	-0.75	100.00%	16.98MiB	754.4KiB	15.35KiB	0	0.0433878	16.85MiB	625.1KiB	12.75KiB	0	0.0362242	False	False
datadog_agent_remap_blackhole_acks	-634.67KiB	-0.98	100.00%	63.1MiB	4.77MiB	99.38KiB	0	0.0756149	62.48MiB	3.46MiB	72.47KiB	0	0.0554403	False	False
splunk_hec_route_s3	-207.3KiB	-1.11	99.77%	18.18MiB	2.35MiB	48.98KiB	0	0.129414	17.97MiB	2.26MiB	47.15KiB	0	0.125509	False	False
syslog_log2metric_humio_metrics	-172.31KiB	-1.27	100.00%	13.22MiB	247.54KiB	5.05KiB	0	0.0182824	13.05MiB	508.85KiB	10.36KiB	0	0.0380667	False	False
http_to_http_acks	-362.11KiB	-1.94	86.57%	18.19MiB	8.55MiB	178.85KiB	0	0.47026	17.83MiB	7.79MiB	162.71KiB	0	0.436768	True	True
datadog_agent_remap_datadog_logs_acks	-1.46MiB	-2.28	100.00%	63.88MiB	3.2MiB	66.95KiB	0	0.0501309	62.42MiB	4.32MiB	89.86KiB	0	0.0691248	False	False
datadog_agent_remap_datadog_logs	-1.81MiB	-2.81	100.00%	64.25MiB	351.78KiB	7.2KiB	0	0.0053454	62.45MiB	4.2MiB	87.43KiB	0	0.067219	False	False
syslog_loki	-465.22KiB	-3.07	100.00%	14.81MiB	621.56KiB	12.72KiB	0	0.040983	14.35MiB	928.29KiB	18.87KiB	0	0.063145	False	False
socket_to_socket_blackhole	-1.4MiB	-5.76	100.00%	24.33MiB	136.59KiB	2.79KiB	0	0.00548152	22.93MiB	254.03KiB	5.19KiB	0	0.0108172	False	False
http_text_to_http_json	-2.78MiB	-7.02	100.00%	39.65MiB	1.04MiB	21.76KiB	0	0.0262466	36.87MiB	942.17KiB	19.24KiB	0	0.0249515	False	False

[github-actions[bot]]

Soak Test Results

Baseline: afcee9be8840174292d47a9e06d19d059f77e246 Comparison: 58faa8dd569351ab85174406a9090e3b4cf98fdd Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	56.06KiB	4.65	100.00%	1.18MiB	118.8KiB	2.42KiB	0	0.0985705	1.23MiB	90.68KiB	1.85KiB	0	0.0718956	False	False
http_pipelines_blackhole	53.78KiB	3.42	100.00%	1.53MiB	39.83KiB	832.97B	0	0.0253471	1.59MiB	102.81KiB	2.1KiB	0	0.063266	False	False
syslog_splunk_hec_logs	152.81KiB	0.95	100.00%	15.77MiB	743.32KiB	15.12KiB	0	0.0460304	15.92MiB	566.02KiB	11.55KiB	0	0.034722	False	False
syslog_humio_logs	113.68KiB	0.67	100.00%	16.52MiB	128.51KiB	2.62KiB	0	0.00759457	16.63MiB	142.57KiB	2.92KiB	0	0.00836938	False	False
syslog_log2metric_splunk_hec_metrics	85.85KiB	0.47	100.00%	17.71MiB	482.42KiB	9.84KiB	0	0.0265951	17.79MiB	788.88KiB	16.05KiB	0	0.0432848	False	False
syslog_regex_logs2metric_ddmetrics	42.9KiB	0.34	99.63%	12.47MiB	504.88KiB	10.29KiB	0	0.0395156	12.52MiB	519.09KiB	10.58KiB	0	0.0404914	False	False
splunk_hec_to_splunk_hec_logs_noack	13.96KiB	0.06	77.44%	23.82MiB	454.04KiB	9.27KiB	0	0.0186086	23.84MiB	335.09KiB	6.84KiB	0	0.0137255	False	False
splunk_hec_route_s3	9.49KiB	0.05	10.94%	18.18MiB	2.36MiB	49.09KiB	0	0.129582	18.19MiB	2.32MiB	48.41KiB	0	0.127274	False	False
splunk_hec_to_splunk_hec_logs_acks	9.74KiB	0.04	31.44%	23.75MiB	854.85KiB	17.39KiB	0	0.0351422	23.76MiB	816.58KiB	16.62KiB	0	0.0335557	False	False
http_to_http_acks	1.9KiB	0.01	0.63%	17.38MiB	8.21MiB	171.57KiB	0	0.472226	17.38MiB	7.98MiB	166.46KiB	0	0.459063	True	True
enterprise_http_to_http	-11.65B	-0	0.12%	23.85MiB	251.18KiB	5.13KiB	0	0.0102846	23.85MiB	255.15KiB	5.22KiB	0	0.0104473	False	False
splunk_hec_indexer_ack_blackhole	-10.92KiB	-0.04	37.02%	23.77MiB	753.73KiB	15.34KiB	0	0.0309563	23.76MiB	819.72KiB	16.68KiB	0	0.0336818	False	False
http_to_http_json	-19.2KiB	-0.08	90.58%	23.85MiB	342.61KiB	6.99KiB	0	0.0140273	23.83MiB	444.36KiB	9.09KiB	0	0.0182078	False	False
file_to_blackhole	-103.53KiB	-0.11	74.11%	95.38MiB	2.16MiB	44.71KiB	0	0.0226098	95.27MiB	3.85MiB	80.04KiB	0	0.0403836	False	False
fluent_elasticsearch	-149.67KiB	-0.18	100.00%	79.47MiB	53.56KiB	1.08KiB	0	0.00065795	79.33MiB	1.35MiB	27.78KiB	0	0.0170121	False	False
http_to_http_noack	-84.33KiB	-0.35	99.94%	23.83MiB	508.39KiB	10.39KiB	0	0.0208303	23.75MiB	1.07MiB	22.37KiB	0	0.0451769	False	False
syslog_log2metric_humio_metrics	-60.97KiB	-0.47	99.97%	12.79MiB	466.37KiB	9.51KiB	0	0.0356034	12.73MiB	694.07KiB	14.12KiB	0	0.0532348	False	False
syslog_loki	-85.55KiB	-0.58	100.00%	14.42MiB	381.94KiB	7.82KiB	0	0.0258563	14.34MiB	761.34KiB	15.48KiB	0	0.0518411	False	False
datadog_agent_remap_blackhole_acks	-662.1KiB	-1.02	100.00%	63.63MiB	3.96MiB	82.52KiB	0	0.0622185	62.99MiB	2.37MiB	49.62KiB	0	0.0376208	False	False
socket_to_socket_blackhole	-429.29KiB	-1.77	100.00%	23.73MiB	398.38KiB	8.13KiB	0	0.0163935	23.31MiB	342.7KiB	7.0KiB	0	0.0143556	False	False
datadog_agent_remap_datadog_logs_acks	-1.16MiB	-1.87	100.00%	61.79MiB	3.2MiB	66.99KiB	0	0.0518574	60.63MiB	4.35MiB	90.55KiB	0	0.071734	False	False
http_text_to_http_json	-791.55KiB	-1.95	100.00%	39.6MiB	794.7KiB	16.22KiB	0	0.019596	38.82MiB	1.08MiB	22.65KiB	0	0.0279064	False	False
http_pipelines_no_grok_blackhole	-291.94KiB	-2.53	100.00%	11.26MiB	57.42KiB	1.17KiB	0	0.00498018	10.97MiB	1.06MiB	22.12KiB	0	0.0968049	False	False
datadog_agent_remap_datadog_logs	-2.06MiB	-3.35	100.00%	61.56MiB	1.33MiB	27.85KiB	0	0.0215736	59.5MiB	4.05MiB	84.35KiB	0	0.068079	False	False
datadog_agent_remap_blackhole	-2.52MiB	-4.32	100.00%	58.42MiB	6.88MiB	143.33KiB	0	0.117779	55.89MiB	4.53MiB	94.56KiB	0	0.0810046	False	False

[github-actions[bot]]

Soak Test Results

Baseline: 36b9243b12a42f0160392b312693e57d81b0a56d Comparison: c065926839838c9542521afbf320ebaa403be122 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	52.36KiB	4.57	100.00%	1.12MiB	114.38KiB	2.33KiB	0	0.0997517	1.17MiB	75.98KiB	1.55KiB	0	0.0633656	False	False
http_pipelines_blackhole	63.37KiB	4	100.00%	1.55MiB	84.4KiB	1.72KiB	0	0.0532533	1.61MiB	131.44KiB	2.68KiB	0	0.0797448	False	False
datadog_agent_remap_blackhole	1.17MiB	1.93	100.00%	60.72MiB	6.31MiB	131.32KiB	0	0.103876	61.89MiB	4.3MiB	89.83KiB	0	0.0695406	False	False
syslog_log2metric_humio_metrics	177.63KiB	1.34	100.00%	12.9MiB	215.46KiB	4.4KiB	0	0.0163027	13.08MiB	474.54KiB	9.67KiB	0	0.0354303	False	False
syslog_regex_logs2metric_ddmetrics	91.47KiB	0.72	100.00%	12.36MiB	574.98KiB	11.71KiB	0	0.0454355	12.45MiB	530.88KiB	10.82KiB	0	0.0416492	False	False
http_to_http_acks	33.11KiB	0.19	11.05%	17.23MiB	8.04MiB	168.15KiB	0	0.466908	17.26MiB	8.09MiB	168.99KiB	0	0.46857	True	True
syslog_splunk_hec_logs	12.21KiB	0.07	40.59%	16.33MiB	774.88KiB	15.76KiB	0	0.046318	16.35MiB	815.89KiB	16.62KiB	0	0.0487337	False	False
splunk_hec_to_splunk_hec_logs_noack	12.08KiB	0.05	72.42%	23.83MiB	430.79KiB	8.8KiB	0	0.0176527	23.84MiB	330.18KiB	6.74KiB	0	0.0135234	False	False
splunk_hec_to_splunk_hec_logs_acks	7.64KiB	0.03	24.83%	23.75MiB	850.94KiB	17.31KiB	0	0.0349827	23.76MiB	828.22KiB	16.85KiB	0	0.0340379	False	False
enterprise_http_to_http	-2.64KiB	-0.01	28.28%	23.85MiB	248.87KiB	5.08KiB	0	0.0101892	23.84MiB	255.94KiB	5.23KiB	0	0.01048	False	False
splunk_hec_indexer_ack_blackhole	-5.48KiB	-0.02	18.10%	23.75MiB	816.18KiB	16.6KiB	0	0.0335529	23.74MiB	847.56KiB	17.23KiB	0	0.0348506	False	False
syslog_humio_logs	-8.9KiB	-0.05	94.49%	16.55MiB	138.34KiB	2.83KiB	0	0.00816009	16.54MiB	179.73KiB	3.68KiB	0	0.0106067	False	False
file_to_blackhole	-60.36KiB	-0.06	54.85%	95.39MiB	2.64MiB	54.7KiB	0	0.0276608	95.33MiB	2.82MiB	58.59KiB	0	0.0295657	False	False
http_to_http_json	-42.5KiB	-0.17	99.85%	23.85MiB	341.05KiB	6.96KiB	0	0.0139634	23.81MiB	561.6KiB	11.46KiB	0	0.0230335	False	False
http_to_http_noack	-85.43KiB	-0.35	99.93%	23.83MiB	521.79KiB	10.67KiB	0	0.0213798	23.75MiB	1.09MiB	22.79KiB	0	0.0460282	False	False
syslog_log2metric_splunk_hec_metrics	-74.31KiB	-0.4	99.97%	18.02MiB	619.51KiB	12.62KiB	0	0.0335617	17.95MiB	798.02KiB	16.24KiB	0	0.043407	False	False
http_text_to_http_json	-192.48KiB	-0.48	100.00%	38.8MiB	1.06MiB	22.19KiB	0	0.0273579	38.62MiB	1.32MiB	27.58KiB	0	0.0341567	False	False
fluent_elasticsearch	-451.56KiB	-0.55	100.00%	79.47MiB	55.03KiB	1.11KiB	0	0.000676123	79.03MiB	4.5MiB	92.44KiB	0	0.056943	False	False
socket_to_socket_blackhole	-167.96KiB	-0.71	100.00%	23.1MiB	204.66KiB	4.18KiB	0	0.00865039	22.94MiB	234.6KiB	4.79KiB	0	0.00998675	False	False
splunk_hec_route_s3	-292.62KiB	-1.56	100.00%	18.36MiB	2.38MiB	49.6KiB	0	0.129689	18.08MiB	2.37MiB	49.47KiB	0	0.130855	False	False
datadog_agent_remap_datadog_logs_acks	-1.09MiB	-1.75	100.00%	61.95MiB	3.65MiB	76.27KiB	0	0.0589166	60.86MiB	4.75MiB	98.77KiB	0	0.0779545	False	False
datadog_agent_remap_blackhole_acks	-1.11MiB	-1.82	100.00%	60.8MiB	5.47MiB	113.93KiB	0	0.0899536	59.69MiB	4.9MiB	102.49KiB	0	0.0821126	False	False
http_pipelines_no_grok_blackhole	-249.03KiB	-2.18	100.00%	11.14MiB	447.18KiB	9.13KiB	0	0.0391813	10.9MiB	1.12MiB	23.3KiB	0	0.102641	False	False
syslog_loki	-342.9KiB	-2.29	100.00%	14.64MiB	344.72KiB	7.06KiB	0	0.0229914	14.3MiB	748.87KiB	15.22KiB	0	0.0511162	False	False
datadog_agent_remap_datadog_logs	-2.01MiB	-3.2	100.00%	62.96MiB	1.97MiB	41.31KiB	0	0.0313	60.94MiB	4.84MiB	100.86KiB	0	0.0794375	False	False