cloud-files
cloud-files copied to clipboard
seung-lab
Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
I encountered some google cloud storage installation issues several times while I was not really using google cloud storage. How about making these dependency optional. Here is [an example of...
e.g. S3 won't allow upload of large files without performing multi-part upload. Related to #7
Not sure if this is the place for asking questions, but I see that for network robustness, when transferring a file, the hash is checked to make sure the destination...
I wonder if we can get a performance benefit by condensing some RANGE requests into multi-part ranges. https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests#multipart_ranges
Hi! I got an email from a guy trying to install `fafbseg` on Google Colab who says install used to work just fine but now just "takes forever". Turns out...
We've currently disabled verification for multi-part, but it seems there's a hidden feature to get the correct information. ```bash curl --head "https://s3.amazonaws.com/bossdb-open-data/iarpa_microns/minnie/minnie65/em/8_8_40/03b8a.shard?partNumber=2" ``` ```bash HTTP/1.1 206 Partial Content x-amz-id-2: +YQe6D4SiKQGldTUf4/mjPxYkTuxrwQ5zVyHcEpL2sd52DvUkzp7xIIOhZflShCVLPPOowBr57o=...
Many object storage services implement the S3 protocol without actually being AWS S3. For example, [Wasabi](https://wasabi.com/wp-content/themes/wasabi/docs/API_Guide/index.html#t=topics%2Fapidiff-intro.htm) and [Open Storage Network](https://www.openstoragenetwork.org/) (OSN). For these services, you define a custom endpoint URL...
It's impossible to do a real rsync command with cloud storage b/c you can't replace pieces of files. However, you can check for whether a file is likely to be...
.boto files are similar to https://docs.python.org/3/library/configparser.html format
HTTP/2
Can we achieve higher performance with HTTP/2? Can we autodetect when it's supported?