Support overriding gcp_cloud_storage API URL (for local dev/test)

[movinfinex]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The GCP Cloud Storage sink currently only supports writing to a real bucket at https://storage.googleapis.com/ It's useful for local development and integration testing to be able to point the sink to some kind of emulator instead. (E.g.: https://github.com/fullstorydev/emulators https://github.com/fsouza/fake-gcs-server)

Specifically, I need some way to make BASE_URL a configurable value, not just a constant.

Attempted Solutions

I think the only alternative is to use something like mitmproxy and get Vector to use a custom cert for https://storage.googleapis.com. I could do this for myself if I had to, but it's difficult to deploy across a team.

Proposal

One option would be to support the Google's own STORAGE_EMULATOR_HOST environment variable. Another option would be something like the endpoint parameter used in the datadog_logs sink.

Either way, I'm willing to write a PR.

References

No response

Version

vector 0.40.0 (aarch64-unknown-linux-gnu 1167aa9 2024-07-29 15:08:44.028365803)