vector
vector copied to clipboard
vectordotdev
New sink: MySQL
Hi, your tool vectordev is really cool, but it lacks of 1 feature which We actively use by using rsyslog, that is output to Postgres and MySQL.
It would be really great if you add such kind of sinks to vectordev.
You can have a look at rsyslog module as an example which We currently use: https://www.rsyslog.com/doc/master/configuration/modules/ompgsql.html
Thank you in advance!
@binarylogic @jszwedko any chance to add this feature in the nearest time? I would extremely be grateful for it!!!
Hi team , we use this great product, but it would be very helpful to use it in end-to-end mode, and postgres support is required for us.
Hi @gartemiev @savadom13 ! Thanks for letting us know! We don't currently have this work planned yet, but knowing that additional users want it will help us prioritize.
I find this feature very useful. Postgres and MySQL are very popular DBs and widely used in companies of different sizes and industries. Having implemented the proposed feature will increase user-base of your product and rating of it as well.
We currently have your product operating in our dev environment - we would like to also deploy in our other environments, if postgres and mysql sinks were supported we could make that move. Thanks for great product.
Yes! We need this feature as well as long as we actively use PostgreSQL for pushing our logs for analytic.
I just want to note that some DB abstraction library should probably be used. The background is to avoid having to maintain DB specific code as much as possible and support additional SQL databases in one go.
I don’t know Rust much, but those two options come up:
- https://docs.rs/odbc/0.17.0/odbc/
- https://github.com/tokio-rs/rdbc
The same applies to #8357. Logstash also does it like that: Jdbc input plugin
Also note the previous discussion about adding a postgres sink in #939.
PS: I am a bit overwhelmed with all the comments basically saying "I need this". That is what the GitHub reactions are for. (Just my personal opinion).
After running into a pretty bad bug in influxdb, we are migrating our metrics to timescaledb. We are quite happy with our migration to vector for log collection and would love to replace telegraf with vector as well. (Telegraf currently reads from Kafka and writes to influx). The only thing stopping us is a postgres/timescaledb sink.
For us it would be fine if this sink strictly handles writing to existing tables, we would actually prefer to handle schema migrations ourselves.
As there is still no SQL sink, there is a workaround with HTTP sink. Run script with http server and write logs to database from there. It might be easily done with Python and FastAPI + SQLalchemy
Repurposing this just for a mysql sink. https://github.com/vectordotdev/vector/issues/939 is tracking a postgres sink.
This would be hugely beneficial to us as well. We love vector and use it a ton. For our use case, until we get MySQL sink, we were thinking of continuing to use vector as an aggregator and use Fluentd sink and then forward things to mysql.
Starrocks should be a good option to sink, it's a new sql and nonsql paradigm very similar to MySQL but using modern in-memory columnar store. The final effect is data source is a MySQL jdbc connection string
I currently use logstash to sync logs between my PG database and OpenSearch. It would be very helpful if Vector could use databases as a data source.
