How to up/download huge files with Feign?

[dibog]

I have to up/download huge files, bigger then fits into memory. For the download I could use the Response object as it gives me the (hopefully) unbuffered input stream. But how could I send big data within a request.

I would like to avoid multipart files or similar. I would like to push the binary data direct behind the request object.

Ideally, I would like to be able to stream data from a response object to a request object without the need to write it into the file system.

Currently, I got stuck with the upload part. The encoder expects that I write the data either as string or byte[] into the RequestTemplate. Is there no way to attach there an output stream?

Thanks, Dieter

[velo]

Hrmm, that is a serious problem... from long time, feign has cache the date into memory using a byte[].

It would be a massive improvement for feign 11 to add the ability to stream data. And breaking some of the compatibility for the next major to include this functionality would be ok

[kdavisk6]

We could consider adding support for a StreamingResponse implementation. Something we would need to design more. @dibog we are open to suggestions. Any thoughts on how you would like this to work?

[dibog]

I had the impression that the Reponse object already allows me to stream the content of the response body without that it was buffered in memory or in the file system.

So what is IMHO missing is a way to fill the request body without it being cached. If I use an Encoder I have to fill the Request.Body object with a byte[]. I would like to store there a reference to an InputStream. That way I'm free to deliver an InputStream implementation of my choice, that could be something in memory, or a FileInputStream, or stream which I got from anywhere else.

[velo]

You are right @dibog , the request loads the contents into memory, being a problem for large payloads.

[dibog]

But the question: Is there a good reason to do that? To compute the Content-Length you would require it, but you can indicate that you don't know the Content-Length. It would be great if the Request mechanism does not expect to get an byte[] but an stream. The system could check if the body is from an bufferable resource (like byte[] or File/Path) or if it is not bufferable.

[velo]

Yeah, we would need to change the encoder to be more a streaming resource instead of accumulating all in a byte[]

[MrSoftwareEngineer]

I was impressed when I opened Request.Body implementation. There is no a workaround excepting different clients using for lightweight request and upload huge files

[lsubbare]

@dibog @velo @kdavisk6 Is the response.body().asInputStream loads whole file into memory? I thought it just streams.

[k-boyle]

Any update on this?

[velo]

Any update on this?

It's still waiting on a community member to raise a PR with a fix.