support for large file upload

[bramverhelst]

The current implementation to save/upload files to S3 uses the put_object boto3 method.

self.s3_connection.put_object(Body=content.read(), **put_params)

Unfortunately, this method has restrictions/thresholds imposed on it via the AWS API, namely a 5Gb size threshold, after which the API will return an error. In order to succesfully upload such big files, one needs to resort to higher level implementations using the client interface (see https://github.com/boto/boto3/issues/1123) OR create the multi-part upload yourself:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

I use the upload_file method myself regularly and this has no issue with such big files :)

[etianen]

If you can submit an MR to implement this, I'd happily take it! I imagine it would improve upload performance too, and avoid streamng the whole file into memory.

[bramverhelst]

Sure. I've submitted a MR.