vectordotdev
add support for sftp source and sink
we have no access right to install vector and mount disk for remote data source. so we can only pull data from remote by sftp.
kafka connect already has a sftp connect.
https://docs.confluent.io/current/connect/kafka-connect-sftp/index.html
@binarylogic @jszwedko Any updates on this? SFTP via a single or pool of ssh connections would be very useful!
This isn't on our near-term roadmap at the moment. We've recently started trying out integrations through OpenDAL, and while not supported yet sftp is on their roadmap.
@spencergilbert Thanks for the info. Looking forward to all this in the future!
@spencergilbert Although not released yet, sftp support was recently merged to OpenDAL :-)
https://github.com/apache/incubator-opendal/pull/2186
SFTP support has been released by OpenDAL: https://github.com/apache/incubator-opendal/releases/tag/v0.33.3
Has there been any updates on this? 🤔
Not that I'm aware of. I think a sink leveraging OpenDAL would be a good start though. If anyone is so motivated you can see the WebHDFS sink for an example of what this might look like (it also uses OpenDAL).
Hi, I start a draft at https://github.com/vectordotdev/vector/pull/18076. Welcome to have a look!
Adding new opendal-based sinks is quite simple, as it mostly involves configuration and documentation tasks.
Hi, I start a draft at #18076. Welcome to have a look!
Adding new opendal-based sinks is quite simple, as it mostly involves configuration and documentation tasks.
Looking forward to this!