python-magic icon indicating copy to clipboard operation
python-magic copied to clipboard

s3 object byte range support

Open tooptoop4 opened this issue 4 years ago • 1 comments

boto3 library has s3 get_object that accepts a byte range (to avoid downloading a whole file but just download selected byte range). if my file is 100mb, can this library do some byte range seek to only download the s3 object partially (ie the first 1000 bytes) and then determine file type?

tooptoop4 avatar Dec 24 '21 16:12 tooptoop4

I'd suggest reading the first chunk of the file (2k or so) and passing that to magic.from_buffer. The underlying libmagic library does have an interface that takes a file descriptor, but I'm not sure how to wire that up to an arbitrary python object.

ahupp avatar Dec 27 '21 19:12 ahupp