Cannot GET entities > 2GB using jdk-connector

[nicobrevin]

It's a bit hard for me to produce a minimal failing test case but I can show you exactly where the problem is.

I am forwarding some large files from openstack swift via jetty, jax-rs-client and the jdk-connector (because the performance seems way way better for big files)

In org.glassfish.jersey.jdk.connector.internal.HttpParser there's a method called decideTransferEncoding where it expects the content length to parse as an int. This fails when the response size is greater than Integer.MAX_VALUE. There might be more issues with this further on down the line, but this was the first one I hit. If I get some time I'll try and get you a patch, but for now I'm going to try and work around it.

[jansupol]

So you use a buffered entity, you buffer over 2GB of data to memory and then you send them to the client. Do I understand the issue correctly?

[jansupol]

Note that for large files, chunked encoding is far better than a buffered stream. Use client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, RequestEntityProcessing.CHUNKED).

[nicobrevin]

@jansupol Thanks for the fix, sorry for the radio silence at the time - This this issue was related to the jax-rs response entity, rather than the request entity - not sure it's possible to force the API we're using to use chunked encoding in its response. But your point stands and I did end up using that for file uploads :+1:

We were not reading the file in to a buffer, we were just proxying it back to a client with something like EntityInputStream#transferTo(ServletOutputStream), i.e. we're using jax-rs to talk to an API and the servlet API to send the file back to our client. For now, we're still using the default connector on the part of our API which is responsible for very large files as we needed to release something and it was working. The async-ness of the jdk-connector was a "nice to have".