feat(new sink): Initial `better_stack_logs` sink

[curusarn]

Initial sink for sending logs to Better Stack - https://betterstack.com/logs

I took another sink that wraps the HTTP sink and modified it. Feedback is very welcome. 🙏

I want to add better_stack_metrics sink next which is the reason for the nested directory structure. I.e. I want to use sinks/better_stack/metrics for the metrics sink so I'm using sinks/better_stack/logs for this sink.

I've put both better_stack_logs and better_stack_metrics behind the same feature gate sinks-better_stack. Configs we provide to people always have both metrics and logs sinks so I don't see a need to separate our sinks under multiple feature gates. Let me know if you want me to change this.

PS: Could you share some advice on how to write the metrics sink? 🙏 I couldn't find a lot of resources on that.

[bits-bot]

All committers have signed the CLA.

[StephenWakely]

Hi. Thanks for contributing.

I'm not sure I see what value this is adding however, since it is possible to just send logs to Better Stack using the http sink. Is there anything that this sink will allow a user to do that isn't covered here?

[curusarn]

Hi @StephenWakely!

Thank you for taking a look at this. I appreciate it!

Custom sink offers multiple advantages over using the HTTP sink.

We've seen backward incompatible changes introduced to the HTTP sink multiple times in the past. Introducing a Better Stack sink will result in having a way to send logs and metrics to Better Stack that is less likely to break with new Vector versions. For example, once, the HTTP sink default method has changed. The result was all Vector configuration for sending logs to Better Stack breaking.

Custom sinks allow us to match better what Better Stack ingesting expects. For example, sometimes logs and metrics arrive without timestamps. With a custom sink, we can add timestamps to events that don't have them. This leads to more accurate timestamps than relying on timestamps added during ingesting. Overall this gives people more accurate and useful data.

Another benefit of a custom sink is simplifying configuration.

Let me know what you think. 🙏

Thanks again for taking a look!

[curusarn]

Hi @StephenWakely,

What do you think about the reasons outlined above?

Does it make sense to you to continue with introducing a Better Stack sink?

Thanks again for taking a look! Let me know what you think. 🙏

[StephenWakely]

Hi. I don't feel that it does make sense to introduce a custom sink here.

Setting the timestamps can be handled with a VRL transform. Generally we don't feel it is necessary to create a custom sink if a work around can be achieved with VRL.

Can you elaborate more on the backward incompatible changes with the HTTP sink? Was it just the changing of the default method, or have there been a significant number of them? Obviously we aim to avoid breaking changes where possible.