Getting a "java.lang.OutOfMemoryError: Java heap space" when upload large (+1 GB) files

[tmoreira2020]

Hi there, after some researching and source code review I identified that there is low support to upload large files. I'm trying to upload a file bigger than 1GB and the following stacktrace pops up on my log.

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at feign.form.multipart.Output.write(Output.java:80)
	at feign.form.multipart.SingleFileWriter.write(SingleFileWriter.java:51)
	at feign.form.multipart.AbstractWriter.write(AbstractWriter.java:36)
	at feign.form.MultipartFormContentProcessor.process(MultipartFormContentProcessor.java:87)
	at feign.form.FormEncoder.encode(FormEncoder.java:105)
	at feign.ReflectiveFeign$BuildFormEncodedTemplateFromArgs.resolve(ReflectiveFeign.java:353)
	at feign.ReflectiveFeign$BuildTemplateByResolvingArgs.create(ReflectiveFeign.java:232)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:84)
	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
	at com.sun.proxy.$Proxy1131.uploadBackup(Unknown Source)

The uploadBackup method is defined like this:

    @Headers("Content-Type: multipart/form-data")
    @RequestLine("POST /backup/upload")
    Backup uploadBackup(@Param("database") File database, @Param("volume") File volume);

I identified that there is low support because the SingleFileWriter.write method loads the entire file into the memory. Are there any other configuration that I can use to fix or workaround this limitation issue?

Thanks

[alecha]

have you found a workaround that is not making the http request manually?

[tmoreira2020]

yes, I used the RestTemplate from Spring Framework to upload the large files, I ended up with two clients implementations 🤦

[alecha]

Thanks @tmoreira2020 , I was able to use RestTemplate to upload large files but unable to make it work for downloading them, did you have the same need?

[vstoyanov]

The trick here (and this applies for any download/upload) is to take advantage of storage media (local hdd, s3, whatever) and keep in memory only a buffer. If keeping a gigabyte file in memory is unstable when running with single client in dev, imagine what's going to happen when there are several parallel downloads on prod? The correct solution is to keep in memory only a buffer and read it from and write it to a stream.