cloudstorage
cloudstorage copied to clipboard
lytics
Cloud & local storage unified api (s3, google, azure, sftp, local)
Hi In order to be possible use yandex S3 (who mimic the S3 API in everything except the endpoint) I'd like to be able to set the endpoint in the...
It would be nice to read file if it already exists in local-file cache, checking md5 to make sure it is the same md5. Use md5 as filename? Would require...
The LocalFS store will not match a filename prefix, it only matches folder prefixes: if a file like `/tests/users-2021-10-12.csv` exists in the base of a LocalFS store. - a query...
Tests were not failing for #66 that aws-s3 did not appropriately handle paging. Before merging in https://github.com/lytics/cloudstorage/pull/67 ensure we have unit tests to catch.
There are `Copy/Move` that are not currently implemented on S3. These are performance optimizations that when copy/move from s3 -> to s3 don't do network transfer, instead allow s3 to...
Many usecases require getting notification immediately after a file is added to a store for import. Support Interfaces for event based file-change-event models. * http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html * https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview * https://cloud.google.com/storage/docs/object-change-notification *...
From this comment https://github.com/lytics/cloudstorage/pull/53#issuecomment-392784874