big request body cause send file error

[chanjarster]

use openresty as a http reverse proxy, upload a file about 1.1M go error describe below.

• image: openresty/openresty:jammy
• nginx version: openresty/1.25.3.2
• nginx.conf in /usr/local/openresty/nginx/conf/nginx.conf (open the debug log):

# nginx.conf  --  docker-openresty
#
# This file is installed to:
#   `/usr/local/openresty/nginx/conf/nginx.conf`
# and is the file loaded by nginx at startup,
# unless the user specifies otherwise.
#
# It tracks the upstream OpenResty's `nginx.conf`, but removes the `server`
# section and adds this directive:
#     `include /etc/nginx/conf.d/*.conf;`
#
# The `docker-openresty` file `nginx.vh.default.conf` is copied to
# `/etc/nginx/conf.d/default.conf`.  It contains the `server section
# of the upstream `nginx.conf`.
#
# See https://github.com/openresty/docker-openresty/blob/master/README.md#nginx-config-files
#

#user  nobody;
#worker_processes 1;

# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;

error_log  logs/error.log  debug;

#pid        logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    # See Move default writable paths to a dedicated directory (#119)
    # https://github.com/openresty/docker-openresty/issues/119
    client_body_temp_path /var/run/openresty/nginx-client-body;
    proxy_temp_path       /var/run/openresty/nginx-proxy;
    fastcgi_temp_path     /var/run/openresty/nginx-fastcgi;
    uwsgi_temp_path       /var/run/openresty/nginx-uwsgi;
    scgi_temp_path        /var/run/openresty/nginx-scgi;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    # Don't reveal OpenResty version to clients.
    # server_tokens off;
}

• provide custom config in /etc/nginx/conf.d/:

upstream up_servers {
    server echo:8080 max_conns=0 max_fails=0;
}
map $http_upgrade $connection_upgrade {
    default Upgrade;
    ''      close;
}
server {
    listen 80;
    client_max_body_size 32m;
    client_header_timeout 30s;
    client_header_buffer_size 4k;
    client_body_timeout 10s;
    large_client_header_buffers 32 64k;
    proxy_connect_timeout 15s;
    proxy_read_timeout 900s;
    proxy_send_timeout 10s;
    proxy_buffers 8 16k;
    proxy_buffer_size 16k;    
    location / {
        proxy_pass http://up_servers;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

• run container:

docker network create --attachable test-network
docker run -d \
                --name echo \
                --network test-network \
                -p 8888:8080 \
                inanimate/echo-server:latest
docker run -d  \
                --name openresty \
                --network test-network \
                -v $(pwd)/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf \
                -v $(pwd)/conf.d:/etc/nginx/conf.d \
                -p 8080:80 \
                "openresty/openresty:jammy"

• upload a file over 1M size:

curl --location --request POST 'http://localhost:8080/' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Accept: */*' \
--header 'Host: localhost:8080' \
--header 'Connection: keep-alive' \
--header 'Content-Type: multipart/form-data; boundary=--------------------------132917510733278170156063' \
--form 'foo=@"/path/to/file"'

• go error below:

2024/11/05 03:07:49 [notice] 1#1: using the "epoll" event method
2024/11/05 03:07:49 [notice] 1#1: openresty/1.25.3.2
2024/11/05 03:07:49 [notice] 1#1: built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 
2024/11/05 03:07:49 [notice] 1#1: OS: Linux 6.1.32-0-virt
2024/11/05 03:07:49 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/11/05 03:07:49 [notice] 1#1: start worker processes
2024/11/05 03:07:49 [notice] 1#1: start worker process 7
2024/11/05 03:08:04 [warn] 7#7: *1 a client request body is buffered to a temporary file /var/run/openresty/nginx-client-body/0000000001, client: 192.168.160.1, server: , request: "POST / HTTP/1.1", host: "localhost:8080"
2024/11/05 03:08:04 [alert] 7#7: *1 sendfile() failed (9: Bad file descriptor) while sending request to upstream, client: 192.168.160.1, server: , request: "POST / HTTP/1.1", upstream: "http://192.168.160.2:8080/", host: "localhost:8080"
192.168.160.1 - - [05/Nov/2024:03:08:04 +0000] "POST / HTTP/1.1" 200 25 "-" "Apifox/1.0.0 (https://apifox.com)"

Did some google and got this:

https://forum.nginx.org/read.php?21,118976,278732#msg-278732

[chanjarster]

@agentzh hello, i will be appreciate if you can take a look at this issue, thank you.

[gavin0616]

@chanjarster，this link https://forum.nginx.org/read.php?21,118976,278732#msg-278732 is 404 now......

In this case,sendfile() failed (9: Bad file descriptor) is related to nginx proxy module, not openresty.

The root cause of this issue is that the server 'inanimate/echo-server' does not properly implement to the rfc7231.

Echo-server sends a 200 response before fully receiving the request body from nginx, which leads nginx to believe that the upstream server (echo-server) no longer needs the request body, so nginx cleans up the locally cached request body file. When the next write event for nginx is triggered, nginx(sendfile function) trys to send the rest part of this file, but this file has already been deleted, then resulting in this error message.

If you want to receive the correct(or full) request body from the echo server, you can try other echo server, like jmalloc/echo-server or echo-nginx-module

The above is my personal understanding, if there are any mistakes, please help to correct them.