403 Forbidden when downloading via MINIO Console or any other S3 tool

[PeterPilley]

NOTE

Re Opening as this is still not resolved Original We have a Minio cluster set running on a docker swarm, we are using traefik to manage the connections. We have a bucket that is a mix of videos and images, the issue we are seeing is we can upload to the nominated bucket fine, however if we try and download the files we are seeing a 403 Forbidden response.

If we create a shared link from the MINIO console the file is available. Which made me think permissions were the issue however the identity we are using has read/write access to the bucket. Do we need to set a seperate policy?

Checking the logs I cannot see any error messages or notifications. Using trace within the Minio Console I can see LockR.lock 403 Forbidden (checking the issues here I can see this is apparently normal?)

Update I dont think this is a permissions issue as using Minio Library within python gives the expected result, using MC gives the expected result. But using MINIO Console and or Cyberduck or s3cmd gives a different result

Expected Behavior

Uploads and Downloads should behave the same with read/write policy

Current Behavior

• I have tried with the same credentials to download the file via mc and it is fine.
• If I try to download via the Minio Console or Cyberduck I get a 403 Forbidden. The account has READWRITE permissions which I understood means they have full get/putObject permissions.
• I have tested all the files uploaded and can actually download some of them but only around 10% the rest present forbidden.
• I have tried a custom access rule but I cannot see any other way to set the "right permissions" so would really appreciate some guidance
• Using the python minio library and can confirm I can upload and download items as expected, the same files when attempting to download via Cyberduck or via Minio Console present Forbidden.
• Also confirmed that using Cyberduck or Minio Console I can create and upload an object but cannot download, I can delete an object or prefix in Minio Console but I cannot delete a prefix in Cyberduck.
• Ran a trace with a blank policy and added in the missing elements
• Results from traces

Refresh Using Cyberduck s3.ListObjectsV1 200 S3.ListMultiPartUpload 200 s3.ListBuckets 200

Attempt download Cyberduck s3.GetObject 403

Attempt download via Minio Python s3.GetBucketLocation 200 s3.GetObject 200

I dont think this is a permissions issue it looks like some files have been allowed to be fetched with s3.GetObject and others haven't even though they were uploaded at the same time with the same identity and were attempting to be downloaded with the same identity

Possible Solution

None Yet

Steps to Reproduce (for bugs)

Create Bucket Add Content Attempt to download content

Your Environment

Docker: Docker version 20.10.17, build 100c701 Configuration: Swarm with 3 managers, 3 workers, Proxy: Traefik Operating System: Ubuntu 20.04

[PeterPilley]

Have just tried again with s3cmd and if I use put instead of sync I can upload

[harshavardhana]

@PeterPilley can you explain what policy was applied on the user can you provide us mc admin policy info output?

[harshavardhana]

I tried with relevant permissions all tools and Console UI work fine. This mainly looks like some policy issue on your end.

[PeterPilley]

@PeterPilley can you explain what policy was applied on the user can you provide us mc admin policy info output?

Hi I only saw this come in now I have tried with full s3* and then also locked down to specifics ie s3 getobject

[PeterPilley]

Hi why was this closed again this hasn't been resolved

[PeterPilley]

I tried with relevant permissions all tools and Console UI work fine. This mainly looks like some policy issue on your end.

What policies? What permissions can you please elaborate. It is not expected that other s3 compliant tools or even the ui would not work and only minio tools will surely.

Is it not possible to troubleshoot this?

[harshavardhana]

What policies? What permissions can you please elaborate. It is not expected that other s3 compliant tools or even the ui would not work and only minio tools will surely.

Unless you can answer the questions that I asked previously https://github.com/minio/minio/issues/15911#issuecomment-1287536659 - there is nothing actionable for us.

[PeterPilley]

What policies? What permissions can you please elaborate. It is not expected that other s3 compliant tools or even the ui would not work and only minio tools will surely.

Unless you can answer the questions that I asked previously #15911 (comment) - there is nothing actionable for us.

I will post the exact policybut I have tried s3:* as well as limiting it to specific permissions (this is also mentioned in my original post). My understanding is s3:* should mean everything is enabled.

What policies and permissions did you add that could be different to that. Can you help me understand why some files will have the ability to be downloaded or even interacted with to get Metadata but not others again I have posted what that looks like in a trace.

[harshavardhana]

Nothing I just used "readwrite" policy that's all everything works as expected.

[PeterPilley]

Nothing I just used "readwrite" policy that's all everything works as expected.

How many files did you try with as mentioned I found some files worked others didn't. Are you in the same version I was using or are you saying on the latest version this is not presenting the same issue.

[harshavardhana]

How many files did you try with as mentioned I found some files worked others didn't. Are you in the same version I was using or are you saying on the latest version this is not presenting the same issue.

We have bazillion production deployments @PeterPilley it is impossible to have such basic issues. I am sure you are doing something funky on your end.

MinIO you must use RELEASE.2022-10-20T00-55-09Z which I tested. There are public deployments running such as https://play.min.io

[PeterPilley]

How many files did you try with as mentioned I found some files worked others didn't. Are you in the same version I was using or are you saying on the latest version this is not presenting the same issue.

We have bazillion production deployments @PeterPilley it is impossible to have such basic issues. I am sure you are doing something funky on your end.

MinIO you must use RELEASE.2022-10-20T00-55-09Z which I tested. There are public deployments running such as https://play.min.io

I do not doubt it, I have also a number deployed which is why it is really weird this is happening. The deployment is the same across all my stacks.

What else could I check that could cause some files to be 403 for getobject and others 200.

[PeterPilley]

Here is the policy I tested when trying to troubleshoot the issue.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:DeleteObject", "s3:GetBucketLocation", "s3:GetObject", "s3:ListAllMyBuckets", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::folder1/", "arn:aws:s3:::folder1/*", "arn:aws:s3:::folder1/image", "arn:aws:s3:::folder1/image/*", "arn:aws:s3:::folder1/video", "arn:aws:s3:::folder1/video/*" ] } ] }

Prior to this it was the default read/write of s3:* with no limit on resource.

[PeterPilley]

RELEASE.2022-10-20T00-55-09Z

I have just redeployed with RELEASE.2022-10-20T00-55-09Z and still can see the same issue

[PeterPilley]

Have tried creating a brand new user with builtin read/write policy, used cyberduck and minio console to try and download files with same result. Some files are able to be downloaded or view the associated metadata others return 403 even within the console