Nginx proxy_pass leading to unit, sometimes outputs 504

[AlekseyArh]

Hi. My application uses unit 1.31.1 with a php embed module.

The client makes POST requests to upload files. Sometimes unit does not respond and nginx proxy issues 504.

php: max_execution_time = 120; unit: idle_timeout, timeout, send_timeout, etc = 130; nginx: proxy_read_timeout = 140;

As a result, there are no entries in the unit logs, but the nginx log has 504 and upstream_response_time 140. At the same time, the CPU is loaded only by 50%.

PS. I came across something similar when uploading using yandex.tank. The processor is free after the end of load testing, but the device still does not respond for several more minutes.

unit.conf

{
    "settings": {
        "http": {
            "server_version": false,
            "discard_unsafe_fields": true,
            "body_read_timeout": 75,
            "header_read_timeout": 45,
            "idle_timeout": 130,
            "log_route": false,
            "max_body_size": 134217728,
            "send_timeout": 130
        }
    },
    "listeners": {
        "*:80": {
            "pass": "routes"
        }
    },
    "applications": {
        "symfony-api": {
            "type": "php8.2",
            "limits": {
                "timeout": 130,
                "requests": 1024
            },
            "processes": {
                "max": 12,
                "spare": 6,
                "idle_timeout": 20
            },
            "targets": {
                "direct": {
                    "root": "/srv/symfony-api/public/"
                },
                "index": {
                    "root": "/srv/symfony-api/public/",
                    "script": "index.php"
                }
            },
            "user": "unit",
            "group": "unit",
            "working_directory": "/srv/symfony-api/"
        }
    },
    "routes": [
        {
            "match":{
                "method": "OPTIONS"
            },
            "action":{
                "return": 204,
                "response_headers": {
                    "Access-Control-Allow-Origin": "${header_origin}",
                    "Access-Control-Allow-Credentials": "true",
                    "Access-Control-Allow-Methods": "DELETE, GET, OPTIONS, POST, PUT",
                    "Access-Control-Allow-Headers": "Accept, Authorization, Content-Type, X-Request-Id"
                }
            }
        },
        {
            "match": {
                "uri": [
                    "*.php",
                    "*.php/*"
                ]
            },
            "action": {
                "pass": "applications/symfony-api/direct",
                "response_headers": {
                    "Access-Control-Allow-Origin": "${header_origin}",
                    "Access-Control-Allow-Credentials": "true",
                    "Access-Control-Allow-Methods": "DELETE, GET, OPTIONS, POST, PUT",
                    "Access-Control-Allow-Headers": "Accept, Authorization, Content-Type, X-Request-Id"
                }
            }
        },
        {
            "action": {
                "share": "/srv/symfony-api/public$uri",
                "fallback": {
                    "pass": "applications/symfony-api/index",
                    "response_headers": {
                        "Access-Control-Allow-Origin": "${header_origin}",
                        "Access-Control-Allow-Credentials": "true",
                        "Access-Control-Allow-Methods": "DELETE, GET, OPTIONS, POST, PUT",
                        "Access-Control-Allow-Headers": "Accept, Authorization, Content-Type, X-Request-Id"
                    }
                },
                "response_headers": {
                    "Access-Control-Allow-Origin": "${header_origin}",
                    "Access-Control-Allow-Credentials": "true",
                    "Access-Control-Allow-Methods": "DELETE, GET, OPTIONS, POST, PUT",
                    "Access-Control-Allow-Headers": "Accept, Authorization, Content-Type, X-Request-Id"
                }
            }
        }
    ]
}

nginx.conf

        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
	proxy_set_header Accept-Encoding "gzip";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_read_timeout 140;
        proxy_intercept_errors on;
        proxy_next_upstream error timeout http_500 http_502 http_503 http_504;
        proxy_next_upstream_tries 2;

[tippexs]

Hi @AlekseyArh

Thanks for reaching out and sorry for the late response!

We are currently looking into several file upload related issues and I would like to include this one in our research. Can you share more about what kind of files and more importantly the size of your files?

We will share updates as soon as we have something new!

Timo

[AlekseyArh]

Hi @AlekseyArh

Thanks for reaching out and sorry for the late response!

We are currently looking into several file upload related issues and I would like to include this one in our research. Can you share more about what kind of files and more importantly the size of your files?

We will share updates as soon as we have something new!

Timo

These are JPG files. The size is approximately 30mb. Photographers upload 100 files sequentially. About 2-5 of them return 504.

Perhaps this is not specifically related to file uploads or POST requests. I did load testing and reduced the load after the peak. nginx+php-fpm woke up quickly. https://overload.yandex.net/661629#tab=test_data&tags=&plot_groups=main&machines=&metrics=&slider_start=1699780045&slider_end=1699780295

unit+php wake up very slowly. https://overload.yandex.net/661630#tab=test_data&tags=&plot_groups=main&machines=&metrics=&slider_start=1699780401&slider_end=1699780651

The feeling that unit accumulates something over time, which does not give it the opportunity to respond until it is cleaned.