vectordotdev
Support ETL use-cases in Vector
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
We've had a number of different requests to support ETL-like use-cases in Vector and so I figured it'd be useful to create this issue to track them all in one place.
Currently Vector is architected for stream processing and doesn't support ETL execution very well. This is primarily due to the lack of source support for bulk execution where the source shuts down after all input has been processed.
Users have asked for this functionality for the file source and the aws_s3 source, but it is easy to see that it could be desirable for any archive-like source. It could even be useful for sources like kafka where it would drain a topic and then shut down.
Refs:
- https://github.com/vectordotdev/vector/issues/7880
- https://github.com/vectordotdev/vector/issues/15609
There is a implementation of batch handling in the file source here: https://github.com/vectordotdev/vector/pull/11667
I presently solve this using stdin and some excessive cat.
(cat logfile; kill -s TERM 0) | ./vector allows me to run a metrics source concurrently while processing the logfile, and exiting vector once the logfile has been completely handled. If you do this inside a bash script, you need to set -m first.
As suggested in some of the links referenced above, I have tried the remove_after_secs: 0 to my file source thinking that may be the use of inotify would induce some behaviour but this did not make any change in the behaviour, I had to Ctrl-C vector to end the process after all files are processed and removed :-(
