fix(gcp sink): Separated healthcheck and regenerate token spawn

[punkerpunker]

Hi there,

This PR is intended to fix the issue mentioned in https://github.com/vectordotdev/vector/issues/13058.

I've moved out auth.spawn_regenerate_token() from the healthcheck_response function and spawned the regeneration token right after healthcheck where it is needed. I was able to build and test the version I am suggesting on my local development k8s cluster and it works as expected - disabled healthcheck doesn't prevent a regeneration token to be spawned.

I am neither familiar with the vector codebase nor with the Rust language, so I'll appreciate any feedback and suggestions and will try my best to make this PR merged, thanks in advance.

@bruceg @spencergilbert

[bits-bot]

All committers have signed the CLA.

[netlify[bot]]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	7ab4cba2913d638428301496b91456596a94445b
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63b6e1cfdaf0ad0008d4044e

[netlify[bot]]

✅ Deploy Preview for vrl-playground ready!

Name	Link
🔨 Latest commit	7ab4cba2913d638428301496b91456596a94445b
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63b6e1cf129f3700071881f4
😎 Deploy Preview	https://deploy-preview-15827--vrl-playground.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[punkerpunker]

LGTM. Does this also apply to the gcp_stackdriver_metrics sink, though?

As far as I see here, auth is being handled at the request level:

https://github.com/vectordotdev/vector/blob/5b471406aad01669776b8934838465b94db73a9b/src/sinks/gcp/stackdriver_metrics.rs#L248

which refers to:

https://github.com/vectordotdev/vector/blob/5b471406aad01669776b8934838465b94db73a9b/src/gcp.rs#L147-L154

In general, this PR is intended to separate healthcheck_response from auth.spawn_regenerate_token(). So, I searched for all usage occurrences of healthcheck_response and applied a fix there. So, stackdriver_metrics.rs doesn't seem to be using this particular function.

[github-actions[bot]]

Regression Test Results

Run ID: 80bb9eee-79e4-464c-b89d-18cb50ff9dda
Baseline: eda64e7cd15054ddc98474783249e88b2a0a9fb9
Comparison: 7ab4cba2913d638428301496b91456596a94445b
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment	Δ mean	Δ mean %	confidence
syslog_humio_logs	-557.11KiB/CPU-s	-5.9	100.00%
syslog_splunk_hec_logs	-612.53KiB/CPU-s	-6.58	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
splunk_hec_route_s3	186.29KiB/CPU-s	1.57	100.00%	11.62MiB/CPU-s	587.11KiB/CPU-s	7.57KiB/CPU-s	0.0	0.049344	11.8MiB/CPU-s	547.49KiB/CPU-s	7.07KiB/CPU-s	0.0	0.045305	False	False
http_to_http_acks	78.38KiB/CPU-s	1.46	87.05%	5.25MiB/CPU-s	2.76MiB/CPU-s	36.49KiB/CPU-s	0.0	0.525644	5.33MiB/CPU-s	2.77MiB/CPU-s	36.62KiB/CPU-s	0.0	0.520118	True	False
http_to_http_json	77.02KiB/CPU-s	0.56	100.00%	13.54MiB/CPU-s	355.68KiB/CPU-s	4.59KiB/CPU-s	0.0	0.025654	13.61MiB/CPU-s	248.47KiB/CPU-s	3.21KiB/CPU-s	0.0	0.017822	False	False
datadog_agent_remap_blackhole_acks	78.1KiB/CPU-s	0.25	99.98%	30.99MiB/CPU-s	1.15MiB/CPU-s	15.21KiB/CPU-s	0.0	0.037112	31.07MiB/CPU-s	1.06MiB/CPU-s	13.96KiB/CPU-s	0.0	0.033979	False	False
splunk_hec_to_splunk_hec_logs_noack	5.72KiB/CPU-s	0.04	85.61%	13.62MiB/CPU-s	232.97KiB/CPU-s	3.01KiB/CPU-s	0.0	0.016703	13.63MiB/CPU-s	194.09KiB/CPU-s	2.51KiB/CPU-s	0.0	0.01391	False	False
http_to_http_noack	4.14KiB/CPU-s	0.03	54.29%	13.61MiB/CPU-s	329.18KiB/CPU-s	4.25KiB/CPU-s	0.0	0.023615	13.62MiB/CPU-s	278.66KiB/CPU-s	3.6KiB/CPU-s	0.0	0.019985	False	False
enterprise_http_to_http	3.99KiB/CPU-s	0.03	74.77%	13.62MiB/CPU-s	224.43KiB/CPU-s	2.9KiB/CPU-s	0.0	0.01609	13.62MiB/CPU-s	149.95KiB/CPU-s	1.94KiB/CPU-s	0.0	0.010747	False	False
splunk_hec_to_splunk_hec_logs_acks	960.52B/CPU-s	0.01	12.04%	13.62MiB/CPU-s	344.34KiB/CPU-s	4.44KiB/CPU-s	0.0	0.024695	13.62MiB/CPU-s	334.11KiB/CPU-s	4.31KiB/CPU-s	0.0	0.02396	False	False
fluent_elasticsearch	-100.33B/CPU-s	-0.0	14.37%	45.41MiB/CPU-s	29.81KiB/CPU-s	389.85B/CPU-s	0.0	0.000641	45.41MiB/CPU-s	30.11KiB/CPU-s	393.84B/CPU-s	0.0	0.000647	False	False
file_to_blackhole	-3.46KiB/CPU-s	-0.01	13.08%	54.5MiB/CPU-s	1.12MiB/CPU-s	14.79KiB/CPU-s	0.0	0.02055	54.49MiB/CPU-s	1.13MiB/CPU-s	14.94KiB/CPU-s	0.0	0.020772	False	False
splunk_hec_indexer_ack_blackhole	-1.75KiB/CPU-s	-0.01	30.34%	13.62MiB/CPU-s	241.63KiB/CPU-s	3.12KiB/CPU-s	0.0	0.017326	13.62MiB/CPU-s	250.45KiB/CPU-s	3.23KiB/CPU-s	0.0	0.01796	False	False
syslog_log2metric_humio_metrics	-7.75KiB/CPU-s	-0.13	84.63%	6.01MiB/CPU-s	222.17KiB/CPU-s	2.87KiB/CPU-s	0.0	0.036106	6.0MiB/CPU-s	357.2KiB/CPU-s	4.61KiB/CPU-s	0.0	0.058126	False	False
otlp_http_to_blackhole	-16.42KiB/CPU-s	-1.03	100.00%	1.55MiB/CPU-s	101.05KiB/CPU-s	1.3KiB/CPU-s	0.0	0.063636	1.53MiB/CPU-s	120.35KiB/CPU-s	1.55KiB/CPU-s	0.0	0.07658	False	False
socket_to_socket_blackhole	-198.31KiB/CPU-s	-1.43	100.00%	13.53MiB/CPU-s	272.82KiB/CPU-s	3.52KiB/CPU-s	0.0	0.019683	13.34MiB/CPU-s	239.54KiB/CPU-s	3.09KiB/CPU-s	0.0	0.017533	False	False
datadog_agent_remap_datadog_logs_acks	-848.3KiB/CPU-s	-2.43	100.00%	34.15MiB/CPU-s	1.16MiB/CPU-s	15.35KiB/CPU-s	0.0	0.034011	33.32MiB/CPU-s	1.0MiB/CPU-s	13.26KiB/CPU-s	0.0	0.030109	False	False
otlp_grpc_to_blackhole	-27.28KiB/CPU-s	-2.55	100.00%	1.04MiB/CPU-s	40.63KiB/CPU-s	537.11B/CPU-s	0.0	0.037993	1.02MiB/CPU-s	54.85KiB/CPU-s	724.57B/CPU-s	0.0	0.052626	False	False
syslog_loki	-244.34KiB/CPU-s	-2.8	100.00%	8.51MiB/CPU-s	402.74KiB/CPU-s	5.2KiB/CPU-s	0.0	0.046222	8.27MiB/CPU-s	244.1KiB/CPU-s	3.15KiB/CPU-s	0.0	0.028823	False	False
datadog_agent_remap_datadog_logs	-1.03MiB/CPU-s	-2.99	100.00%	34.4MiB/CPU-s	1.2MiB/CPU-s	15.9KiB/CPU-s	0.0	0.034978	33.38MiB/CPU-s	1.44MiB/CPU-s	18.97KiB/CPU-s	0.0	0.043017	False	False
syslog_regex_logs2metric_ddmetrics	-134.72KiB/CPU-s	-3.56	100.00%	3.7MiB/CPU-s	468.45KiB/CPU-s	6.05KiB/CPU-s	0.0	0.123609	3.57MiB/CPU-s	431.29KiB/CPU-s	5.57KiB/CPU-s	0.0	0.118	True	False
syslog_log2metric_splunk_hec_metrics	-360.84KiB/CPU-s	-3.73	100.00%	9.45MiB/CPU-s	253.45KiB/CPU-s	3.27KiB/CPU-s	0.0	0.026181	9.1MiB/CPU-s	243.95KiB/CPU-s	3.15KiB/CPU-s	0.0	0.026176	False	False
datadog_agent_remap_blackhole	-1.2MiB/CPU-s	-3.82	100.00%	31.31MiB/CPU-s	1.01MiB/CPU-s	13.41KiB/CPU-s	0.0	0.03238	30.11MiB/CPU-s	1.36MiB/CPU-s	17.96KiB/CPU-s	0.0	0.04511	False	False
http_text_to_http_json	-1.05MiB/CPU-s	-4.23	100.00%	24.93MiB/CPU-s	620.22KiB/CPU-s	8.01KiB/CPU-s	0.0	0.024292	23.88MiB/CPU-s	595.16KiB/CPU-s	7.68KiB/CPU-s	0.0	0.024339	False	False
syslog_humio_logs	-557.11KiB/CPU-s	-5.9	100.00%	9.23MiB/CPU-s	234.87KiB/CPU-s	3.03KiB/CPU-s	0.0	0.024851	8.68MiB/CPU-s	173.95KiB/CPU-s	2.25KiB/CPU-s	0.0	0.019559	False	False
syslog_splunk_hec_logs	-612.53KiB/CPU-s	-6.58	100.00%	9.09MiB/CPU-s	169.83KiB/CPU-s	2.19KiB/CPU-s	0.0	0.018252	8.49MiB/CPU-s	460.92KiB/CPU-s	5.95KiB/CPU-s	0.0	0.053027	False	False