S3-Performance-Test
S3-Performance-Test copied to clipboard
Upload very big files
At the moment, when uploading a file, the file content is generated in the heap memory prior to the Put Object request: https://github.com/jenshadlich/S3-Performance-Test/blob/master/src/main/java/de/jeha/s3pt/operations/Upload.java#L39
This limits the file size to the maximum heap size (-Xmx) divided by the number of threads. With -Xmx1G
and 4 threads, I can not send files bigger than 256M without generating an OOME.
2 solutions are possible, and both seem interesting to me:
- Generate file content as it is uploaded. It should be possible to implement an InputStream which randomly generates bytes as they are requested.
- Generate file content as a local temporary file. Even if the time taken to write the local file is a pure loss, it would allow S3 client to use a "send file" system call which is more efficient (no Java heap required).