Document required IAM permissions for operations

[rafalkrupinski]

First of all, I want to thank all the developers here for this project. It's really great help!

I'd like to ask to improve the documentation to mention permissions required by s3fs operations. #532 shows that they can be surprising sometimes.

I have

"Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:CreateBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::bucket",
                "arn:aws:s3:::bucket/*",

and still can't save a file 🤷

[martindurant]

If you figure it out, it would indeed make a good addition to the docs. You should look at the specific request that is being denied and compare against the API doc pages which ought to state what you need, e.g., this for CreateMultipartUpload: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html

[orf]

@rafalkrupinski try PutObject* and GetObject*. Also try enabling debug logging on s3fs (set S3FS_LOGGING_LEVEL=debug) and putting the logs here?

[rafalkrupinski]

It seems that writing a file requires ListBucketV2 grant to the bucket and PutObject to the path, at least for a small file (no multipart). I had both grants. Problem was in code that was granting the permissions (https://github.com/aws/aws-cdk/issues/22060). Workaround was to create a separate policy.

S3FS could use the documentation of required grants anyway, so I'm not closing the issue :)

[rabidaudio]

In the meantime, in case anyone stumbles across this issue, it seems this one is also required:

        {
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }

Hopefully no one else looses as much time as I did on that one

[rafalkrupinski]

I don't have it and it works :)

[martindurant]

Some of these depend on what you are doing! ListAllMyBuckets would get used when you do fs.ls("") to get a list of buckets associated with your account. If you already know which bucket(s) to use, you never need that.

[rafalkrupinski]

@martindurant Sorry, I didn't get that you're referring to the problem of documenting the required privileges, and not just the particular problem I've encountered 🤷🏻

[martindurant]

I am in complete agreement with you, the privileges should be documented as well as we are able. I'm just saying that not all of them are needed by a user, depending on the situation, so that documentation needs to be carefully written. Would you like to have a go?

[rafalkrupinski]

Maybe there could be some automated testing facility that would take a mapping between s3fs function call and a policy, set up a couple of buckets accordingly and run the calls against them? I'm not sure how to implement it in details, but AWS CDK runs Python, tests could be local or run in lambdas.

BTW, the privileges I had were right, the problem was with CDK with silently doesn't apply any changes to resources that were imported from outside the CloudFormation stack using ARN. Also, the default no_create=False in open() made it extra confusing, bc the error was about missing privilege to create a bucket.

[martindurant]

We do not test against read AWS S3 at all, but against a local server provided by moto. That means we are not able to fully test the permissioning model at all.