remote_file.read does not read full file for >1GB files

[cschull]

I am trying to read a 2GB+ files in chunks. However, when I read the file in chunks with remote_file.read(1024102450), in order to return chunks of 50MB at a time, it times out at the 3rd method call and only reads 25MB instead of 50MB. All following remote_file.read calls returns a chunk size of 0.

I initially tried using the Down.download method to download the files locally but for files larger than 2GB, the full file size was not being downloaded (downloading a 2GB resulted in a file size of 1.6GB and downloading a 5GB file returned a local file size of 2.3GB and 1.4GB when trying a second time). There are many inconsistencies using the download method.

I am using the exact code that is used in the documentation. Does this library have a solution for large file sizes or is there a way for it to prevent timeouts when streaming large files?

[janko]

Hmm, is there a file I can test this with? Timeout errors should usually be problems with the server, or with the client not setting timeouts high enough (which is the responsibility of users of this gem).

[cschull]

Hi Janko, thanks for your reply. Upon further investigation I think you're right. I believe it is an issue with the server. I got the download to work using an http client that forces http2, which protocol version does the Down gem use?

[janko]

I believe Down uses HTTP 1.1, because AFAIK neither net/http nor http.rb clients support HTTP 2. I would like to add a HTTPX backend in the near future, but I don't know when that would happen.

[cschull]

Ok, thanks for the clarification

[janko]

Just pinging @HoneyryderChuck to let him know there is an interest in the HTTPX backend for Down 😉

[HoneyryderChuck]

That's great! Let me know if you need me to review the PR.