couchdb icon indicating copy to clipboard operation
couchdb copied to clipboard

Provide a way to export a database to a file and to any S3 compatible storage

Open nkosi23 opened this issue 3 years ago • 0 comments

Summary

The recently open-sourced yandex-db database has a feature enabling users to export a database to both the file system and any S3-compatible storage. The import story is similarly easy.

It would be great to have the same feature in couchdb, specifically to help with the replication of very large databases (hundreds of gigabytes, dozens of terabytes or even petabytes) to a different data center. Transmitting everything over the wire is impractical in these situations. In the 1.X era, one simply needed to take a snapshot of the .couch file, but with clustering there is no clear solution to this problem.

Desired Behaviour

Users should be able to both export and import and a database in one click from Fauxton or using curl. The feature only needs to support the export of a whole database (no need to support partial exports), and the import of the whole database into a new database. Supporting imports to an existing database would unnecessarily introduce the problem of conflict handling. For simplicity, users can simply import to a new database and realize filtered replications as needed to a local database.

Additional context

Sometimes it is needed to move or replicate very large databases to a different data center not necessarily having a 10gbps internet link. This feature would help facilitate the transfer of the bulk of data, while newest data will be able to be transferred using the replication protocol.

nkosi23 avatar May 29 '22 22:05 nkosi23