Reverse Proxy with MinIO Server availability problem

[chainyo]

Hey, Proxy Manager is really really awesome, it saves me a lot of time but I am facing a little issue with my personal single mode deployment MinIO server.

I have deployed on my Synology NAS (DS415+) a MinIO server though Docker, here is the command (these are not the real root user credentials obviously):

docker run -d --name minio  \
    -p 9100:9000 -p 9101:9001 \
    -e "MINIO_ROOT_USER=root" \
    -e "MINIO_ROOT_PASSWORD=pass" \
    -e "MINIO_DOMAIN=s3.chainyo.tech" \
    -v /volume1/docker/minio/data:/data \
    -v /volume1/docker/minio/config:/root/.minio \
    quay.io/minio/minio server /data --console-address ":9101"

Everything is running well in local, I can access the console (192.168.1.x:9101) and the API (192.168.1.x:9100) though Python script.

But I am using Ngninx Proxy Manager to make this service available outside my local network with my own domain name which is chainyo.tech.

So I added the minio container in the proxy network to make them communicate.

I have setup two redirections:

• Console: on s3.chainyo.tech I can access to the console on port 9101, everything is running well, it's working I can access to the console, I can login and manage the server without any restrictions.
• API: on s3.api.chainyo.tech I can't use this endpoint in my Python script, the client seems to connect but finally timeout after a long long time.

Here is my simple test script, which timeout after the "client connected" print:

from minio import Minio

def connect(endpoint, access_key, secret_key):
    minioClient = Minio(endpoint, access_key=access_key, secret_key=secret_key, secure=False)
    
    return minioClient

client = connect("s3.api.chainyo.tech:9100", "root", "pass")
print("client connected")

buckets = client.list_buckets()
for bucket in buckets:
    print(bucket.name, bucket.creation_date)

bucket = "public-bucket"
object_to_push = "test.txt"

client.fput_object(bucket, "test.txt", object_to_push)
print("Uploaded object to minio")

And finally this is my Nginx conf file inspired from the MinIO documentation I have found here:

    server {
        set $forward_scheme http;
        set $server         "minio";
        set $port           9100;

        listen 80;
        listen [::]:80;

        server_name s3.api.chainyo.tech;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;

		# Block Exploits
        include conf.d/include/block-exploits.conf;

        access_log /data/logs/proxy-host-15_access.log proxy;
        error_log /data/logs/proxy-host-15_error.log warn;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio:9100
        }
    }

This is the error trace while trying to interact with the api in Python:

Traceback (most recent call last):
  File "/home/chainyo/code/minio/upload_to_minio.py", line 12, in <module>
    buckets = client.list_buckets()
  File "/home/chainyo/.local/lib/python3.8/site-packages/minio/api.py", line 633, in list_buckets
    response = self._execute("GET")
  File "/home/chainyo/.local/lib/python3.8/site-packages/minio/api.py", line 397, in _execute
    return self._url_open(
  File "/home/chainyo/.local/lib/python3.8/site-packages/minio/api.py", line 266, in _url_open
    response = self._http.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 330, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='s3.api.chainyo.tech', port=9100): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd11dc65e20>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Is there anything I am missing ? Anyone have already achieved that kind of stuff ?

P.S. I used ports 9100 and 9101 because I'm already using port 9000 for portainer service.

[chainyo]

No one ever deployed a MinIO server ? 😢

[rickmills]

Hit the same issue, here's how I ended up getting it working:

• Enable HTTP/2 in the 'SSL' section
• Make sure your api subdomain is pointing to whatever the equivilent of port 9000 is. So on the example in your code above that would be port 9100

[chainyo]

Hit the same issue, here's how I ended up getting it working:

• Enable HTTP/2 in the 'SSL' section
• Make sure your api subdomain is pointing to whatever the equivilent of port 9000 is. So on the example in your code above that would be port 9100

Hey thanks for your answer, I understand the SSL section part, but what do you mean by API subdomain ? @rickmills

[rickmills]

So I found I basically had to have two domains. 1 for the web ui and one for anything interacting with minio from a script or command line. It has to be a separate domain pointing at port 9000.

One other step I missed out, in Nginx proxy manager make sure you've ticked the box to enable 'Websockets support' as this will force proxy manager to forward any host headers, which are needed for most applications.

[chainyo]

So I found I basically had to have two domains. 1 for the web ui and one for anything interacting with minio from a script or command line. It has to be a separate domain pointing at port 9000.

OK! Good to know! Thanks for your insights!

[samgaw58]

So I found I basically had to have two domains. 1 for the web ui and one for anything interacting with minio from a script or command line. It has to be a separate domain pointing at port 9000.

OK! Good to know! Thanks for your insights!

@ChainYo Did you ever get this working? I have a minio docker image running, Nginx Proxy Manager has two sub domains pointing at ports 9001 (web console) & 9002 for the block storage. I want to be able to backup a Synology NAS using HyperBackup. If I set HyperBackup up pointing at the sub domain on port 9002 I can connect as far as creating a new bucket, but it then informs me that the connection has been dropped. If I use your settings above edited to match my own settings. The Proxy host immediately goes offline once I save the config.

[chainyo]

So I found I basically had to have two domains. 1 for the web ui and one for anything interacting with minio from a script or command line. It has to be a separate domain pointing at port 9000.

OK! Good to know! Thanks for your insights!

@ChainYo Did you ever get this working? I have a minio docker image running, Nginx Proxy Manager has two sub domains pointing at ports 9001 (web console) & 9002 for the block storage. I want to be able to backup a Synology NAS using HyperBackup. If I set HyperBackup up pointing at the sub domain on port 9002 I can connect as far as creating a new bucket, but it then informs me that the connection has been dropped. If I use your settings above edited to match my own settings. The Proxy host immediately goes offline once I save the config.

I decided to use Cloudflare Tunnels. That's so easier!!

[samgaw58]

OK thanks, someone somewhere must have gotten this to work!

[stilet]

OK thanks, someone somewhere must have gotten this to work!

Create 2 domains and create 2 entries with different ports: dev.test.com - console, files.test.com - server. Add MINIO_SERVER_URL=https://files.test.com in docker enviroment

[Adigezalov]

@ChainYo Hello. Did you manage to solve this problem? I ran into exactly the same problem and can't seem to find a solution to the problem.

[mathiznogoud]

MINIO_SERVER_URL=https://files.test.com

legend. worked for me

[zandhaas]

I still have the same issue after trying all of the above.

@mathiznogoud. Is it possible you share your configuration of the nginx PM and the docker compose with us?

[scottruzal]

I got this working after setting the MINIO_BROWSER_REDIRECT_URL environment variable in Docker.

See documentation here and below: https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html

You must also set the following environment variables for the MinIO deployment:

Set the MINIO_BROWSER_REDIRECT_URL to the proxy host FQDN of the MinIO Console (https://example.net/minio/ui)

[tech1-baps]

So I found I basically had to have two domains. 1 for the web ui and one for anything interacting with minio from a script or command line. It has to be a separate domain pointing at port 9000.

OK! Good to know! Thanks for your insights!

@chainyo Did you ever get this working? I have a minio docker image running, Nginx Proxy Manager has two sub domains pointing at ports 9001 (web console) & 9002 for the block storage. I want to be able to backup a Synology NAS using HyperBackup. If I set HyperBackup up pointing at the sub domain on port 9002 I can connect as far as creating a new bucket, but it then informs me that the connection has been dropped. If I use your settings above edited to match my own settings. The Proxy host immediately goes offline once I save the config.

I decided to use Cloudflare Tunnels. That's so easier!!

@chainyo i am also trying to use Cloudflare Tunnels but i am not able to get the MinIO Console Page to load.

[tech1-baps]

So I found I basically had to have two domains. 1 for the web ui and one for anything interacting with minio from a script or command line. It has to be a separate domain pointing at port 9000.

OK! Good to know! Thanks for your insights!

@chainyo Did you ever get this working? I have a minio docker image running, Nginx Proxy Manager has two sub domains pointing at ports 9001 (web console) & 9002 for the block storage. I want to be able to backup a Synology NAS using HyperBackup. If I set HyperBackup up pointing at the sub domain on port 9002 I can connect as far as creating a new bucket, but it then informs me that the connection has been dropped. If I use your settings above edited to match my own settings. The Proxy host immediately goes offline once I save the config.

I decided to use Cloudflare Tunnels. That's so easier!!

did you use Cloudflare Tunnels completely in place of Ngninx? or in combination with Ngninx?

[chainyo]

So I found I basically had to have two domains. 1 for the web ui and one for anything interacting with minio from a script or command line. It has to be a separate domain pointing at port 9000.

OK! Good to know! Thanks for your insights!

@chainyo Did you ever get this working? I have a minio docker image running, Nginx Proxy Manager has two sub domains pointing at ports 9001 (web console) & 9002 for the block storage. I want to be able to backup a Synology NAS using HyperBackup. If I set HyperBackup up pointing at the sub domain on port 9002 I can connect as far as creating a new bucket, but it then informs me that the connection has been dropped. If I use your settings above edited to match my own settings. The Proxy host immediately goes offline once I save the config.

I decided to use Cloudflare Tunnels. That's so easier!!

did you use Cloudflare Tunnels completely in place of Ngninx? or in combination with Ngninx?

Hi @tech1-baps tbh I don't really remember precisely the manipulation I did to make it run as it was a long time ago I did that.

If I remember well I dropped totally nginx for cloudflare tunnels as it re-route the service correctly, but the cloudflare documentation is always evolving so I can't give you the exact manipulation I did.

[xrh0905]

Hi. From my test, it appers that Host is the problem of this issue. If manually override the proxy_set_header Host $host; into proxy_set_header Host $http_host;, the minio start to respond as expect. Hope this get fixed soon since it's impossible to overwrite this head from UI nor custom configs but have to overwrite /etc/nignx/conf.d/proxy.conf since proxy_set_header cannot be re-set.

[CodFrm]

I started using docker-compose and then mounted the modified proxy.conf successfully.

add_header       X-Served-By $host;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Scheme $scheme;
proxy_set_header X-Forwarded-Proto  $scheme;
proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP          $remote_addr;
proxy_pass       $forward_scheme://$server:$port$request_uri;

version: '3.8'
services:
  app:
    image: nginx-proxy-manager:2.11.1
    restart: unless-stopped
    entrypoint: bash -c "pip install zope -i https://pypi.tuna.tsinghua.edu.cn/simple && exec /init"
    ports:
      - '8000:80'
      - '8002:81'
      - '8001:443'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - ./proxy.conf:/etc/nginx/conf.d/include/proxy.conf

[wudingjian]

具体解决方法：详见官方文档：https://www.minio.org.cn/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html

如果你需要部署对象存储的网站是https，需要在Nginx Proxy Manager里将9080端口进行反代，并进行端口映射。例如端口映射到99，那你的endpoint就变成了https://域名:99

# Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      proxy_pass http://minio:9000;
      }