cottoncandy
cottoncandy copied to clipboard
gallantlab
sugar for s3
* Roundtrip test for non-continguous F and C order arrays that trigger at 2GB -- as per defaults * Roundtrip test for all dtype/order conditions that use large array compression...
With @gxlilyBerkeley, we identified a bug when storing raw arrays, which was quite annoying. If we upload some arrays and download them immediately, they might be changed in the process....
Rewrite `upload_npy_array` and `download_npy_array` to do in-memory and have a npz compressed option Reference: https://github.com/numpy/numpy/blob/master/numpy/lib/format.py
Something along the lines of: https://github.com/mumrah/s3-multipart
I am trying to use cottoncandy to upload my numpy arrays to s3 and this code is getting executed inside AWS EMR spark. But while running getting the below errors:-...
There are situations in which the system-wide `defaults.cfg` should be preferred over the user configuration options (`options.cfg`). There should be an option during install to select which configuration is preferred....
should be very straight forward: https://cloud.google.com/storage/docs/boto-plugin#uploading-objects
I'm having trouble ls'ing the root of a bucket. might be a boto/botocore issue. the problem is that `os.path.normpath` turns '' into '.', and then botocore barfs. for ref, boto...
This needs to be cleaned. It's super confusing at the moment. This goes for variable names and methods. I also have the feeling that there is an unnecessary layer of...