afero-s3
afero-s3 copied to clipboard
ability to read file partially
hi!
currently the package can only open a stream from the opened position -> end of file
i store indexed blobs on s3 and perform small random reads on them (static database), so for me, it is useful to open a stream and read only say, 1kb out of a 30gb file
i made this change which makes Read() only request up to the size of the byte buffer passed in. I also added tests related to this, and the current tests pass (at least on my computer)
this does mean that current applications which use readAll on a large file, the request will be broken up into many smaller requests. Whether or not this is preferable, it is not for me to say. The previous behavior can be replicated through passing in a byte buffer with a length larger than or equal to the size of the file, something which is known to the user before they have a chance to call read.
i'm currently using a fork of this library for this purpose, would love to be able to switch back. let me know if there is anything else i would need to do to get this in!
hi @fclairamb i saw the activity on this
so i actually stopped work on this branch and wrote a new package combining stuff from this package, another package, and my fork. https://gfx.cafe/open/s3fs/-/blob/master/file.go
more than happy to move a bunch of that work into a new clean fork from your main branch, let me know.