vector
vector copied to clipboard
vectordotdev
Make the `http` sink `uri` option templateable
Currently, the http sink's uri option is static. It would be nice to make it templateable, allowing users to partition data across different uris. Ex:
[sinks.my_http_sink]
type = "http"
uri = 'https://localhost/{{application_id}}
@LucioFranco assigning this to you since this should be much easier now, correct? It has a decently high user value. If this is a big change please unassign yourself and we can talk about it more.
This would require some changes to our httpsink trait and update our http sink to be partition able. So I think this may be a bit harder than simply adding a templatable uri.
@bruceg is this doable? And is it somewhat related to your batching work?
@LucioFranco is right that this is more difficult than just adding templating to the URI, since it means collecting multiple batches simultaneously. I think we already have the infrastructure to do this, but that doesn't mean it will necessarily be a simple change. It does touch on batching behavior, but should be mostly independent of the batching rework.
Having this feature would help vector to write nicely into fluentd which currently has the message tag as part of the url path: https://docs.fluentd.org/input/http
Having this feature would help vector to write nicely into fluentd which currently has the message tag as part of the url path: https://docs.fluentd.org/input/http
As an aside, I suspect we will be adding a dedicated sink for fluent, now that we have the source in place - but I'm not sure on the timeline for that
That sounds great as well, but in the interim the HTTP transport works well enough :) I'd also question if it is worth to add a fluentd "forward" protocol sink when HTTP works just as well (aside from the tag thingy in the URL)? While I appreciate adding support for more protocols I think that if the other side speaks HTTP well enough, why not just advertise that instead. Granted the forward protocol might exhibit some better behavior with ACKs and the likes, so it might still be worth to add.
That sounds great as well, but in the interim the HTTP transport works well enough :) I'd also question if it is worth to add a fluentd "forward" protocol sink when HTTP works just as well (aside from the tag thingy in the URL)? While I appreciate adding support for more protocols I think that if the other side speaks HTTP well enough, why not just advertise that instead. Granted the forward protocol might exhibit some better behavior with ACKs and the likes, so it might still be worth to add.
Definitely 🙂 there's also work being done around codecs and decoders, so the source/sink may be a layer of "generic http sink + known codec" and for ease of use the user just says "fluent" and we chain the necessary bits under the covers.
Any update here? we'd like to use http-sink as our webhook out. To achieve this, we need the entire URL as a template, not just the path.
Another 👍🏻 for this. I asked for this feature via discord, eventually pointed ehre :)
If only I knew rust.
https://discord.com/channels/742820443487993987/1126537900549492736/1126537900549492736
quick update: I've got it fully working in my local! Just tests & documentation remaining and will submit it by end of week
quick update: I've got it fully working in my local! Just tests & documentation remaining and will submit it by end of week
Thanks, looking forward to this contribution!
