Slow S3 performance (MiniO)

[DEPSTRCZ]

Current Behavior

This might be unrelated, but I wanted to at least try to get some insights into this issue. Since i am really out of ideas

I am using a MinIO SNSD deployment for Pterodactyl backups and have observed strangely slow performance when MinIO is accessed through an Nginx HTTPS reverse proxy. Specifically, a 5GB part upload takes approximately 7 minutes. However, when I configure the panel to use S3 via direct IPv4:port, the upload completes in only about 3 minutes.

To me, this suggests that the issue might lie with the Nginx proxy. To investigate further, I conducted the same test using the MinIO client, comparing uploads through the Nginx proxy and direct IPv4. For this test, I used a 5GB file to simulate the specified part size in Pterodactyl and included the --disable-multipart flag to simulate Wings sending one part at a time. The results were consistent across both IPv4 and the Nginx proxy, with upload times of approximately 3 minutes in both cases. This means the problem might not lie in the Nginx proxy.

Notes

• The machine that serves the Nginx Proxy nad MiniO has enough of available resources.
• The Nginx Proxy is not under Cloudflare.
• Multiple part sized were tested.
• Nginx Proxy before MiniO version: 1.18.0

Expected Behavior

Faster upload speeds. Without any slow downs.

Steps to Reproduce

1. Set up a remote MinIO SNSD deployment.

2. Configure an Nginx reverse proxy for the MinIO deployment.

   • Test the speed of Wings backups when accessed through the Nginx proxy.

3. Test the speed of Wings backups via direct IPv4 access.

   • Bypass the Nginx proxy and directly access MinIO using its IPv4:port configuration. Compare the upload performance with the proxy test.

Panel Version

1.11.10

Wings Version

1.11.13

Games and/or Eggs Affected

None

Docker Image

None

Error Logs

No ordinary logs are generated.

Is there an existing issue for this?

• [X] I have searched the existing issues before opening this issue.
• [X] I have provided all relevant details, including the specific game and Docker images I am using if this issue is related to running a server.
• [X] I have checked in the Discord server and believe this is a bug with the software, and not a configuration issue with my specific system.

[danny6167]

Thank you for the detailed submission. Just for completeness sake, could you please provide the version of nginx and the nginx config.

[DEPSTRCZ]

Of course. It is: 1.18.0

Here is the config in its current state..

server {

   listen       443;
   listen  [::]:443 http2;
   server_name  admin-ui.example.com;


    # SSL Configuration
   ssl_certificate /etc/nginx/certs/origin.pem;       # Cloudflare self-signed certificate
   ssl_certificate_key /etc/nginx/certs/origin.key;   # Cloudflare private key
   ssl_protocols TLSv1.2 TLSv1.3;
   ssl_prefer_server_ciphers on;
   ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";
   ssl_session_timeout 1d;
   ssl_session_cache shared:SSL:10m;
   ssl_session_tickets off;


    # Security headers
   add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
   add_header X-Content-Type-Options nosniff;
   add_header X-Frame-Options DENY;
   add_header X-XSS-Protection "1; mode=block";
   add_header Referrer-Policy "no-referrer-when-downgrade";
   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-NginX-Proxy true;

      # This is necessary to pass the correct IP to be hashed
      real_ip_header X-Real-IP;

      proxy_connect_timeout 300;

      # To support websocket
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      chunked_transfer_encoding off;

      proxy_pass http://localhost:9001/; # This uses the upstream directive definition to load balance
   }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl http2;
    server_name  s3-endpoint.example.com;

    # SSL Configuration
    ssl_certificate /etc/letsencrypt/live/s3-endpoint.example.com/fullchain.pem;   # Certbot certificate
    ssl_certificate_key /etc/letsencrypt/live/s3-endpoint.example.com/privkey.pem; # Certbot private key
#    ssl_protocols TLSv1.2 TLSv1.3;
#    ssl_prefer_server_ciphers on;
#    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
#    ssl_session_timeout 1d;
#    ssl_session_cache shared:SSL:10m;
 #   ssl_session_tickets off;
    

    # Allow special characters in headers
    ignore_invalid_headers off;  
    

    # Limit request body size
    client_max_body_size 0;

    # Disable buffering
    proxy_buffering off;
    proxy_request_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://localhost:9000; # This uses the upstream directive definition to load balance
    }
}



server {
    listen 80;
    listen [::]:80;
    server_name admin-ui.example.com;

    # Redirect all HTTP traffic to HTTPS
    return 301 https://$host$request_uri;
}
server {
    listen 80;
    listen [::]:80;
    server_name s3-endpoint.example.com;

    # Redirect all HTTP traffic to HTTPS
    return 301 https://$host$request_uri;
}

[theshiper]

did anything come up yet with what the slow down could be?

[whytf]

Ahoj, I would recommend you trying out HAproxy, as your problem is most likely related to requests and/or overhead with reverse proxy tech.

Requests comparison: https://youtu.be/h-ygQbBROXY?si=nKIz9CxgxPnLhzsO&t=227 Latency comparison: https://youtu.be/h-ygQbBROXY?si=pXt66oEKd-_TobwG&t=260

More in depth: https://youtu.be/o4gjiBetlZw?si=aDJN669WDFUHVzWl&t=477 In your case focus on HTTPS comparison