dolt icon indicating copy to clipboard operation
dolt copied to clipboard

Ensure ability to use AWS S3 compatible data stores

Open fire opened this issue 5 years ago • 6 comments

S3 is not just AWS's protocol but a protocol for many other systems. Dolt's support for S3 compatible apis would be great.

fire avatar Apr 01 '20 18:04 fire

Like you want to be able to push to a remote from Dolt that has an S3 compatible API? And you want that supported in the remote syntax? Can you give us an example of one you want that we can build to?

timsehn avatar Apr 01 '20 18:04 timsehn

https://www.s3-client.com/s3-compatible-storage-solutions.html

Here's a list.

I would start with https://min.io/

fire avatar Apr 01 '20 18:04 fire

Dolt's S3 storage requires Dynamo as well, to store manifest contents. In order for S3 compatible storage offerings to work, we would need to implement a file-based s3 manifest, as is currently the case with GCS remotes.

zachmu avatar Feb 06 '23 21:02 zachmu

@timsehn Any update on this?

omar391 avatar Jun 05 '23 01:06 omar391

@timsehn Any update on this?

We support S3, filesystem, GCS, and a custom remote server called remotesrv. We don't have much demand for more supported remote types.

https://docs.dolthub.com/sql-reference/version-control/remotes

timsehn avatar Jun 05 '23 14:06 timsehn

just +1'ing here as someone exploring dolt. I'd love a more generic s3/ MinIO compatible remote (supposing this would have to be added as a new nbs/blobstore). This doesn't look overly difficult to write given the others already implemented and would be a huge value.

priley86 avatar Jun 16 '25 14:06 priley86