vector
vector copied to clipboard
vectordotdev
New sink: postgres
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Use Cases
Users want to write data to Postgres from Vector.
Breaking this off from https://github.com/vectordotdev/vector/issues/939
Attempted Solutions
No response
Proposal
No response
References
- https://github.com/vectordotdev/vector/issues/6556
- https://github.com/vectordotdev/vector/issues/939
Version
vector 0.26.0
This is certainly an interesting kind of sink: databases. I wonder if it would make more sense to implement an ODBC sink instead of specific database sinks? This way, any database that provides ODBC drivers (there are many already) could interface with Vector out of the box. In addition, probably would be much less maintenance burden for Vector than maintaining multiple specific sinks.
Agreed, supporting something like ODBC would let us hit a lot of them at once. https://github.com/pacman82/odbc-api seems promising as a Rust wrapper.
AFAIK, rsyslog implements both approaches: ODBC based and database-specific based. The first one allows using with the same plugin for as many databases as possible without database-specific features. The second one uses database-specific features.
I was also going to suggest a more general sink, at least to start with
Unless I am missing something what is needed for Postgres specifically is already in the current Cargo.toml (according to the PRs on this repo anyway)..
https://github.com/vectordotdev/vector/pull/18391/files
My suggestion would be to have something that addresses a specific set of (potentially limited) use cases. For me, having the ability to write to Db (a specific table) avoids having 1 more thing in the middle like Redis/Rabbit/etc.,
This is a fair thing to do for a limited number of things received (logs, metrics, messages whatever). When I say limited I am talking 100K+/day -- that is enough for a good set of use cases. Hopefully nobody receiving 100K/min is going to write to a db :)
