self-hosted
self-hosted copied to clipboard
getsentry
Logs flooded with client request body is buffered to a temporary file messages
Version
21.21.0
Steps to Reproduce
- have some app write their data
docker-compose logs --tail=30 -tf nginx
Expected Result
clean logs ;)
Actual Result
[warn] 23#23: *24007 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000010755, client: 192.168.24.21, server: , request: "POST /api/54/store/ HTTP/1.1"
I think /var/cache/nginx/client_temp/0000010755 should at least be stored on a volume. writing to image paths directly is ugly and likely slow (overlayfs layers come to play).
Here's an example how to increase buffers:
- https://github.com/eventum/docker/commit/58c06696f42c6ad76f4f2fa7e517fbf1df1ac1c4
Yep. client_temp is on overlay:
/var/cache/nginx/client_temp # df -h .
Filesystem Size Used Available Use% Mounted on
overlay 4.9G 3.6G 1.1G 77% /
21.21.0
Month 21, eh? :P
For reference here are the docs for nginx proxy_buffering.
But we seem to have this off by default? 🤔
https://github.com/getsentry/self-hosted/blob/48dd5eb144858b47c7631dffcd325021c4daa7a2/nginx/nginx.conf#L39
Helped for me to set the client_body_buffer_size to 16k
--- a/nginx/nginx.conf
+++ b/nginx/nginx.conf
@@ -84,6 +78,9 @@
}
location ~ ^/api/[1-9]\d*/ {
proxy_pass http://relay;
+
+ # https://github.com/getsentry/self-hosted/issues/1381
+ client_body_buffer_size 16k;
}
location / {
proxy_pass http://sentry;
- https://serverfault.com/questions/511789/nginx-client-request-body-is-buffered-to-a-temporary-file
By default sentry's nginx is fine with proxy_buffering off; and it should better be off anyway as cached responses are unnecessary and cannot help IMO.
Do you see a reason to include this in default configuration?
proxy_buffering is irrelevant here, you need client_body_buffer_size to remove those warnings:
- https://github.com/getsentry/self-hosted/issues/1381#issuecomment-1068121074
For future reference: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size
Do you think this should be documented somewhere? Add it to be configured using environment variable? or something else?
why not up the value? or at least add the default to confg for easy reference
why not up the value? or at least add the default to confg for easy reference
I agree with adding the default value commented to nginx.conf.
16k still too small for client_body_buffer_size, upping to 32k:
2022-06-12T14:14:51.796262685Z 2022/06/12 14:14:51 [warn] 22#22: *6540365 a client request body i
s buffered to a temporary file /var/cache/nginx/client_temp/0002571913, client: 192.168.24.73, server: , request: "POST /api/54/store/ HTTP/1.1
", host: "sentry.example.org"
Hi, this is still relevant. For us, when the sentry is under higher load, it starts buffering requests to temp file and (i believe it is related, nginx starts responding with 400, yet i have to debug what is response of that 400) and the logs got flooded (quite a lot) with:
[28/Jan/2024:11:13:17 +0000] "POST /api/2/envelope/ HTTP/1.1" 400 115 "-" "sentry.php/3.22.1" "168.119.14.43"
2024/01/28 11:13:16 [warn] 22#22: *29308 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000035226, client: 172.18.0.7, server: , request: "POST /api/2/envelope/ HTTP/1.1", host: "sentry.xyz.dev"
[28/Jan/2024:11:13:16 +0000] "POST /api/2/envelope/ HTTP/1.1" 400 115 "-" "sentry.php/3.22.1" "168.119.14.43"
2024/01/28 11:13:16 [warn] 22#22: *29308 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000035225, client: 172.18.0.7, server: , request: "POST /api/2/envelope/ HTTP/1.1", host: "sentry.xyz.dev"
[28/Jan/2024:11:13:16 +0000] "POST /api/2/envelope/ HTTP/1.1" 400 115 "-" "sentry.php/3.22.1" "168.119.14.43"
2024/01/28 11:13:16 [warn] 22#22: *29308 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000035224, client: 172.18.0.7, server: , request: "POST /api/2/envelope/ HTTP/1.1", host: "sentry.xyz.dev"
[28/Jan/2024:11:13:15 +0000] "POST /api/2/envelope/ HTTP/1.1" 400 115 "-" "sentry.php/3.22.1" "168.119.14.43"
2024/01/28 11:13:15 [warn] 22#22: *29308 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000035223, client: 172.18.0.7, server: , request: "POST /api/2/envelope/ HTTP/1.1", host: "sentry.xyz.dev"
Sentry version 24.1.0
