Avoid unnecessary head_object requests

[neg3ntropy]

I am using boto to download "folders" containing lots of small files from S3. I have noticed that an extra HEAD request is being performed before each GET, which results in slowdowns, throttling and $$$.

I understand that knowing the size beforehand is required to to decide whether to do a multipart download or not, but I consider the current behavior to be wasteful, because:

• I might disable multipart but the request is still performed
• I might already know the size of the object (from the list_objects that I am iterating on)

I propose supporting a "size hint" parameter to bypass the head_object call.