pyfilesystem icon indicating copy to clipboard operation
pyfilesystem copied to clipboard

Using StringIO/BytesIO instead of tempfiles with S3 filesystem?

Open shoyer opened this issue 9 years ago • 0 comments

I tried using pyfilesystem's s3fs to process a large quantity of data from Amazon's S3 (more than fits on disk) and I found that operation failed, because I ran out of space for the temp files s3fs creates.

Would it make sense to have an option or change the default to load files directly into memory instead of writing them to disk?

shoyer avatar Jul 30 '15 22:07 shoyer