Enable immediate NGINX proxy cache streaming

[akarasulu]

Problem

Big files are a big problem. In the following configuration, on a cache MISS the client has to wait until the cache node pulls down the whole file from the upstream server:

[client] <-> [cache node] <-> [upstream origin]

We will be referencing vagrant files in our CDN. These files are very large. It is important to have the cache nodes returning something on a cache miss even before the whole file is pulled down from the upstream server.

Possible Solutions

There are a number of NGINX modules and their options we SHOULD consider. Most notably we need to look at the slice plugin.

Also, we should probably use a cache lock to prevent multiple clients from requesting the same file. This makes the proxy start downloading from the upstream several times and slows everything down. Here's a page that talks about this: https://www.nginx.com/blog/smart-efficient-byte-range-caching-nginx/

Also, consider changing the use_temp_path option to the proxy_cache module to off. Find out where NGINX puts temp files. In a LEDE router that I installed NGINX on NGINX pushed to /tmp and killed the router. Need to turn this off I think.