documentation icon indicating copy to clipboard operation
documentation copied to clipboard

Disable proxy buffering (download big files)

Open melroy89 opened this issue 5 years ago • 5 comments

Disable proxy buffering with Nginx reverse proxy, in order to download large files without issues. As stated here:

https://docs.nextcloud.com/server/stable/admin_manual/configuration_files/big_file_upload_configuration.html?highlight=big%20files

Currently added to the server context, any feedback is welcome if you think it needs only be added to a certain location context instead.

Regards, Melroy

melroy89 avatar Dec 09 '20 01:12 melroy89

cc @josh4trunks @jivanpal

MorrisJobke avatar Jan 15 '21 21:01 MorrisJobke

On bigger headers requests, you maybe want to set proxy_buffer_size, proxy_busy_buffers_size and proxy_buffers as well. So the end result will be:

proxy_buffering off; # Pass _receive_ requests directly to the server

proxy_buffer_size 12k; # Increase first part of the responsive, which contains headers.
proxy_busy_buffers_size 16k; # Therefore, also increase the busy buffer response that is not yet fully read by the client
proxy_buffers 64 4k; # 64 times 4k response buffer

Response data is always buffered. So proxy_buffering off does not invalidate the proxy_buffer_size settings.

melroy89 avatar Jan 18 '21 21:01 melroy89

Do large downloads/uploads work on Apache without tweaking, or do you need to use tweaks from the "Uploading big files" page of the manual? Just trying to confirm why they need to be in the main nginx configuration and not just be explained in the "Uploading big files" page?

Can we link to some example of Issues that would we be solved by this addition? Would be helpful to see how common it is needed. The only thing I have set in my production configuration is client_max_body_size 2G; and I believe anything up to 2GB works; but maybe I'm not hitting the same issue as others for some reason.

josh4trunks avatar Jan 19 '21 12:01 josh4trunks

Do large downloads/uploads work on Apache without tweaking, or do you need to use tweaks from the "Uploading big files" page of the manual? Just trying to confirm why they need to be in the main nginx configuration and not just be explained in the "Uploading big files" page?

client_max_body_size is set in this "Nginx configuration" page, although maybe still too low for some people.

You are right, fastcgi_read_timeout isn't set nor explained on this page.

The problem I try to solve initially with this PR is about:

By default, downloads will be limited to 1GB due to proxy_buffering and proxy_max_temp_file_size on the frontend.

See docs


Can we link to some example of Issues that would we be solved by this addition? Would be helpful to see how common it is needed.

It may solves: https://github.com/nextcloud/server/issues/21472 as an example.

The only thing I have set in my production configuration is client_max_body_size 2G; and I believe anything up to 2GB works; but maybe I'm not hitting the same issue as others for some reason.

Be aware, you are mixing upload & download Nginx configurations! client_max_body_size is about uploads to the server. proxy_buffering is about downloads from the server. These are different problems.

melroy89 avatar Jan 19 '21 12:01 melroy89

Thanks for explaining my misunderstanding with client_max_body_size pertaining to uploads and not downloads. But my point is that I have none of these settings (so they would set to nginx defaults) and I just confirmed that a 2.3GB file downloaded just fine.

But, maybe my setup isn't hitting any timeouts since it is on my LAN. In this case, someone else would be better to test and confirm that this fixes large file downloads with a standard nginx setup.

josh4trunks avatar Jan 19 '21 12:01 josh4trunks