console Minio console session error when exposed on subpath

Minio console session error when exposed on subpath

Open realies opened this issue 2 years ago • 6 comments

Unable to expose the console on an nginx reverse proxy subpath using minio/minio:RELEASE.2022-07-08T00-05-23Z.

  minio:
    container_name: minio
    command: ["server", "/data", "--console-address", ":9001"]
    image: minio/minio:latest
    environment:
      - MINIO_SERVER_URL=http://localhost:9000
      - MINIO_BROWSER_REDIRECT_URL=http://localhost/console
    ports:
      - 9000:9000

		location /console/ {
			rewrite ^/console/(.*) /$1 break;
			proxy_pass http://minio:9001;
		}

Getting a 401 response on http://localhost/console/api/v1/session while everything else appears to load up fine:

{"code":401,"message":"unauthenticated for invalid credentials"}

Jul 11 '22 07:07 realies

https://github.com/minio/console/issues/1908

I have the same problem So, now I use RELEASE.2022-06-17T02-00-35Z (here ) and can touch to login to the website and view the console However, the object download is creating an XMLRequest with a static path So you can't download the object (it will hang....)

https://github.com/minio/console/blob/df38c84075f9926c381c0cbb1631ee51dbd468a9/portal-ui/src/screens/Console/Buckets/ListBuckets/Objects/utils.ts#L20-#L49

Also, when logging out Its redirection is also #/subpath/subpath/login

After the RELEASE.2022-06-17T02-00-35Z Image Tag Will got the same issue like your figure

Jul 11 '22 09:07 j13tw

I have attempted to use RELEASE.2022-06-17T02-00-35Z as well, but I cannot even login. It seems that every time I try to login, I get the following error: "Expected element type <AssumeRoleResponse> but have <html>"

Is it maybe because i have my console setup on https://domain.com/s3-console and my bucket on https://domain.com/s3 ?

From my nginx logs:

nginx_1    | 2022-07-16T16:19:22.368763358Z 000.000.00.00 - - [16/Jul/2022:16:19:22 +0000] "POST /s3-admin/api/v1/login HTTP/1.0" 500 118 "https://domain.com/s3-admin/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36" "-"

What really confuses me, is that I can hit other endpoints just fine. I can get {"loginStrategy":"form"} from the /login endpoint, for instance.

Jul 16 '22 16:07 djnnvx

I too am not able to expose minio console on subpath using latest version and also above mentioned versions (with nginx reverse proxy in front).

Jul 19 '22 15:07 mskyttner

I'm here to explain the practices and limits I set up for success Currently using Kubernetes + Istio to build VS implementation set ENV MINIO_BROWSER_REDIRECT_URL Add "http://your.specify.domain/console" "/console/" to specify console leads to "/" (port 9001) The original MinIO S3 API uses "/" (port 9000) In my solution, the console and MinIO s3 API are used same domain

Solve the problem that the Console download Object is abnormal Need to refer to the #1908 settings

Jul 20 '22 00:07 j13tw

@j13tw Thanks for reaching out and it sounds like you confirm that there is a workaround available? I tried the following nginx configuration, which I think does what you suggest:

server {

	charset UTF-8;
	ignore_invalid_headers off;
	client_max_body_size 0;
	proxy_buffering off;

	location /minio-console {
            rewrite   ^/minio-console/(.*) /$1 break;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;
	    proxy_pass http://minio:9001;
        }


	location / {
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header Host $http_host;

		proxy_connect_timeout 300;
		# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
		proxy_http_version 1.1;
		proxy_set_header Connection "";
                chunked_transfer_encoding off;
		proxy_pass http://minio:9000;
	}

}

The docker-compose.yml has the MINIO_BROWSER_REDIRECT_URL set:

  minio:
    image: quay.io/minio/minio:latest
    volumes:
      - ./minio-share:/data/d1
    command: server --console-address=":9001" /data/d1
    environment:
      - MINIO_ROOT_USER=secret
      - MINIO_ROOT_PASSWORD=secret
      - MINIO_DOMAIN=data.bibliometrics.lib.kth.se
      - MINIO_SERVER_URL=https://data.bibliometrics.lib.kth.se
      - MINIO_BROWSER_REDIRECT_URL=https://data.bibliometrics.lib.kth.se/minio-console
#      - CONSOLE_SUBPATH=/minio-console
      - MINIO_API_SELECT_PARQUET=on
    healthcheck:
      test: ["CMD", "curl", "-f", "http://minio:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    restart: always

The result is that when I go to the domain where the minio server is running, I get a redirect to that domain with "/minio-console/" added, which stops due to a "401" as mentioned in the original issue here, since /minio-console/api/v1/session returns a JSON respone with code 401 and message "unauthenticated for invalid credentials":

The same "401" appears when using minio/minio:RELEASE.2022-06-17T02-00-35Z. With this version (everything else being the same) the redirect when hitting the minio server URL is "doubled" and instead goes to "/minio-console/minio-console/login". When manually fixing this and requesting "/minio-console/login" instead , I can see the login screen but not login with my credentials:

I think there is a main.js javascript POST happening to /minio-console/api/v1/login and the error message is {"code":500,"detailedMessage":"Post \"https://data.bibliometrics.lib.kth.se/\": dial tcp 127.0.1.1:443: connect: connection refused","message":"invalid Login"}.

This error mentions an IP and port which I'm quite sure I haven't configured anywhere...

I'm rolling back my attempt to expose minio console on a subpath using nginx reverse proxy in front, until I have a better understanding of how/if this is possible with or without a workaround.

Thankful for any more details/hints for reproducible workarounds.

Jul 20 '22 08:07 mskyttner

Is it maybe because i have my console setup on https://domain.com/s3-console and my bucket on https://domain.com/s3 ?

If you set the api under the path https://domain.com/s3, it could cause a routing issue to ingress when the console try to do STS login with WebIdentity, the request path starts /?Action=AssumeRoleWithWebIdentity... which nginx is unable forward traffic to the API server.

172.18.0.1 - - [21/Jul/2022:17:41:57 +0000] "POST /?Action=AssumeRoleWithWebIdentity&DurationSeconds=3600&Version=2011-06-15&WebIdentityToken=e<my-token> HTTP/1.1" 302 138 "-" "Go-http-client/1.1" 1064 0.000 [-] [] - - - - 1eefe30fd6a8f786fb9f8c54fa0bdd68

Here are logs found in my ingress, as you could see there is no destination ([-]) service shown (marked as bold).

add this environment variables could be helpful.

CONSOLE_MINIO_SERVER=https://domain.com/s3 which basically set STS endpoint.

code

func GetSTSEndpoint() string {
	return strings.TrimSpace(env.Get(ConsoleMinIOServer, "http://localhost:9000"))
}

Jul 21 '22 18:07 hsinhoyeh

This actually works already in all situations

worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
        server minio4:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }

        location /minio {
            rewrite   ^/minio/(.*) /$1 break;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
    }
}

version: '3.7'

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2022-08-13T21-54-44Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
    - "9001"
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
    MINIO_BROWSER_REDIRECT_URL: http://10.0.0.185:9000/minio
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - data2-1:/data1
      - data2-2:/data2

  minio3:
    <<: *minio-common
    hostname: minio3
    volumes:
      - data3-1:/data1
      - data3-2:/data2

  minio4:
    <<: *minio-common
    hostname: minio4
    volumes:
      - data4-1:/data1
      - data4-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:

So whatever is not working is perhaps generally a setup issue here. Also please use /minio instead of some custom names here. /minio is a reserved prefix that can be used here.

Aug 23 '22 05:08 harshavardhana

So whatever is not working is perhaps generally a setup issue here. Also please use /minio instead of some custom names here. /minio is a reserved prefix that can be used here.

I'm using RELEASE.2023-04-07T05-28-58Z. You SHOULD NOT use /minio. Otherwise mc admin will not work. Because mc admin will access /minio/admin.

Apr 07 '23 18:04 sudoexec

Doesn't work in Kubernetes with the nginx config posted by @harshavardhana.

Exposed locally as nodeport (30090 nodeport -> 8080 nginx -> 9001 console):

{
    "code": 401,
    "detailedMessage": "Post \"http://localhost:30090/\": dial tcp 127.0.0.1:30090: connect: connection refused",
    "message": "invalid Login"
}

Exposed via nginx-ingress on customer's cluster (custom PKI):

{
    "code": 401,
    "detailedMessage": "Post \"https://minio.customer.com/\": x509: certificate signed by unknown authority",
    "message": "invalid Login"
}

I need a option to tell the console where the minio instance is reachable (in this case simply http://minio:9000).

Update

Fixed by not setting MINIO_SERVER_URL (or setting it to http://localhost:9000). Confusing that the official documentation tells me to set it to the external fqdn.

Aug 13 '23 19:08 peterhirn

@peterhirn the referenced documentation is for Linux, where hosts would typically have some sort of externally-accessible FQDN.

You are on K8s, so there are an entirely different set of expectations around FQDN of pods/services.

We do not at this time intend to write detailed documentation on nginx reverse proxies in K8s. That's two steps removed from MinIO, and we don't have the resources to test/validate it against the myriad of possible K8s + DNS configurations out there.

We're discussing adding something of a disclaimer at the top guiding users to look at that reference as a baseline that may/will require modification, vs something guaranteed to work in a broad sense.

Aug 14 '23 16:08 ravindk89

@peterhirn Thanks a lot for the hint about MINIO_SERVER_URL. It ended a long hour of uncertainty and frustration for me. I was setting up a MinIO server behind Caddy as a reverse proxy. Everything looked fine, but the login failed, as described above. I did set MINIO_SERVER_URL because it was recommended by the documentation and it made sense, but that was the issue. I removed it, and now everything works fine.

@ravindk89 I have full understanding that the documentation cannot cover all cases, but currently it says that MINIO_SERVER_URL maybe necessary if "The Console must use a specific hostname to connect or reference the MinIO Server, e.g. due to a reverse proxy or similar configuration.". This sounded like my case, and MINIO_SERVER_URL is mentioned in many examples online, so I set it, but it broke my setup. The troubleshooting is very hard then. Maybe the documentation could be more specific, or at least mention that this variable should not be set unless some specific problems occur.

Dec 13 '23 18:12 ocroquette

Maybe the documentation could be more specific, or at least mention that this variable should not be set unless some specific problems occur.

That's actually how we got to where we are now. Without that guidance, we had users who were frequently running into issues with the Console failing to load or redirecting incorrectly when reverse proxies were configured.

It would be easier if the errors were always the same for a given set of inputs, but that hasn't been my/our experience. It varies depending on MinIO version, K8s setup, nginx setup, and whether or not you're using Operators/custom charts/official charts/hand-written YAML.

I actually think this behavior overall might improve in newer versions of MinIO, as we are refining how the Console and Server talk to each other. I have to check how necessary this is now.

Dec 13 '23 18:12 ravindk89

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }

        location /minio {
            rewrite   ^/minio/(.*) /$1 break;
            proxy_set_header Host $http_host;

@harshavardhana Thank you so much for your complete answer. Just a quick update: If you want to access the minio api in /minio subpath and console UI in /console subpath, you should add the following line to the Nginx reverse proxy configuration:

	location /minio/ {
		rewrite   ^/minio/(.*) /$1 break;
		proxy_pass http://minio/;

		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_connect_timeout 300;
		proxy_http_version 1.1;
		proxy_set_header Connection "";
		chunked_transfer_encoding off;
	}

        location /console {
		rewrite   ^/console/(.*) /$1 break;
		proxy_pass http://console/;

		proxy_http_version 1.1;
		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-NginX-Proxy true;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		real_ip_header X-Real-IP;
		proxy_connect_timeout 300;
		chunked_transfer_encoding off;
	}

Apr 06 '24 22:04 mmhlego

console console copied to clipboard

Minio console session error when exposed on subpath

console
console copied to clipboard