meltano
feat: generic S3 tap
Problem:
AWS S3 is by far the most popular object storage out there. Yet Meltano won't let me retrieve (or put, in a hub-supported way) most of my files.
Typical use cases would be to retrieve files in the "usual formats" including:
- csv (that one is already covered by one tap)
- json
- jsonl
- parquet
- ...
The Airbyte S3 connector does a good job at this:
Solution? I would love to have one tap where I can simply select all these things and be on my way.
Notes Since this workflow might turn out to be rather maintainance intensive, this would make a great candidate for a tap to be taken over by Meltano (as also mentioned in the issue regarding this topic).
Would be great to have a storage provider. Which could be aws, azure, gcp blob storage systems
@sbalnojan we have https://github.com/MeltanoLabs/Singer-Most-Wanted/issues?page=1&q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc as a list of most wanted connectors. What are your thoughts on moving this issue there?
@tayloramurphy I don't mind moving, but I do think it should be easy to get a list of the feature requests already out there, shouldn't it? Seems like plugin feature requests now land here, whereas tap/target requests land in over in the "Singer Most Wanted" (btw. "Most Wanted" kind of makes it feel weird to create something just "I" want).
fwiw: https://github.com/MeltanoLabs/Singer-Most-Wanted/issues/37 there already is a target request similar to this one.
@sbalnojan the main complicating factor on this is that we store the metadata for the connectors here but most everything else related to taps and targets is in MeltanoLabs.
I think in my ideal world we'd have the Hub repo be about Hub specific bugs/features and features for connectors specifically would go elsewhere. But perhaps that's more confusing for the community.
Either way, we need to get https://github.com/meltano/hub/issues/754 up so we can have placeholders on the Hub itself!