caddyserver
reverseproxy: Rewrite request buffering for fastcgi
Follow up on 6637. Now this patch will try to buffer request body for fastcgi requests by default with memory and file based buffers. The buffering behavior can be disabled and the buffer limits can be set as well.
This makes sure caddy can work out of the box when the requests don't have a content length, typically chunked encoding, without having to configure request buffering, which will buffer all requests and will not set content-length if the request can't be buffered in memory. content-length is required when communicating with php-fpm.
If buffering is disabled, response for requests without content-length will be 411 Length Required, typically for http1 requests that have chunked encoding. If the buffer limit is reached, the response will be 413 Request Entity Too Large.
By default, memory buffer is 16k per request, and all file based buffers can only use 100MB of disk space. The buffer file path is dependent on the system, and caddy wil fail to start if caddy can't create files in this directory.
The new configuration parameters are following:
body_buffer_disabled
body_buffer_memory_limit <size>
file_buffer_size_limit <size>
file_buffer_filepath <path>
Another fix is that now content-length is checked when sending the request to php-fpm, so that request size is limited to prevent client to send faulty content-length.
Also, longer term we should think about generalizing the buffering so it could be used by any reverse_proxy transport. There are cases where it could be useful for some users, like being able to retry a POST on a different upstream (rewind and resend the body) or to tee it to two upstreams at once (e.g. to send it to a new version of an upstream to compare the result with the old one -- not sure if buffering is needed for that but it could help?)
Is this superseded by #6661? If so, maybe we can close this.
(Thanks for all your help in fixing the problem, Weidi!)
It might be slightly more difficult to encourage Caddy use in the PHP space if buffering is completely off the table. The standard bearer reverse proxy in that space does buffer to the filesystem for FastCGI by default. But, I can also appreciate that buffering is much more complicated.