renku-python
renku-python copied to clipboard
Writable S3 storage mounts
As a Renku user, I want to have write access to a mounted S3 dataset, so that data that is added to the dataset, is also copied to the remote S3 storage.
Acceptance Criteria
- Data added (i.e.
renku dataset add <dataset> <URI>
) to a dataset with a mounted S3 backend, is also copied to the S3 storage.
Notes
- Checksum calculation should be consistent with the remote S3.
- ATM, we don't have a writable S3 storage for testing this feature.
Having a read/write mount makes it really easy to corrupt the source data (e.g. rm -r data/
would data all data in the remote storage if dataset was mounted).
We should discuss if we want to have this feature or at least enable it by explicit request from users (e.g. --read-write-mount
).