public-cloud-roadmap
public-cloud-roadmap copied to clipboard
ovh
Object Storage - bucket policies
As a customer, I need to set S3 bucket policies. As an example, I need to set a public access for all my objects in my bucket.
API: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html
On Object Storage S3, High Performance and Standard Performance
In S3 there 2 ways to defined permissions on a bucket: via ACLs (legacy but not deprecated) and bucket policies using AWS IAM syntax.
/!\ important ACLs and bucket policies are not mutually exclusive i.e you can use ACLs and bucket policies at the same time, however, please note that the principle of least privilege applies and can summerized as "allow only if there is an explicit allow and no explicit deny, else, deny all"
We will provide an S3 compatible REST endpoint to put, delete, dowload a policy configuration file that respects the AWS IAM policy syntax for a given bucket.
The following configuration elements will be supported:
- statement definition with support for "effect", "principal", "action" and "resource" definition
- ip whitelisting with conditional keys
The following configuration elements will not be supported:
- AWS IAM roles will not be supported as it is a feature specific to AWS IAM
- "groups" in "principal" defintion (a global overhaul of the identity management system in the OVH ecosystem is discussed)
- conditional keys (for now): as it is an advanced functionality of policies, do not hesitate to submit an improvement request and upvote it if you want it to be put in the pipeline (except for IP whitelisting)
There is a s3 policy import function, im not quite sure how its working. Importing a policy does not change anything.
Already available are user policy (not yet bucket policy) You will read example of S3 User policy and the structure of the json here : https://docs.ovh.com/ie/en/storage/s3/identity-and-access-management/
Yeah we tried the user policy. We imported an JSON but it didn't and no error is seen. I try again with this example. Thanks.
@larsonnn Same here, no effect when adding a user policy for a bucket. Want to restrict, upload by this user to BUCKET_B. First gave read-write to BUCKET_A, nothing. Restricted BUCKET_B to read only, nothing. Can still upload to BUCKET_B without any issues using multer-s3 tool.
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListMultipartUploadParts",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::BUCKET_A",
"arn:aws:s3:::BUCKET_A/*"
],
"Sid": "RWContainer"
},
{
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:ListMultipartUploadParts",
"s3:ListBucketMultipartUploads"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::BUCKET_B",
"arn:aws:s3:::BUCKET_B/*"
],
"Sid": "ROContainer"
}
]
}
Have tried to make this work for 6 hours straight... The policies have no effect. Wtf is OVH doing releasing functionnalities like that... At least add a "beta" or "wip" label in the UI damn it !
Also I can always list the swift/openstack based buckets. This is a problem because you can't assign a user in "DENY" mode to this kind of buckets...
It is my understanding that public access on AWS S3 is granted with a bucket policy (cf. https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-use-case-2) and that we can't have the same level of access on an entire Object Storage bucket until this functionnaliity is implemented. Any idea of an ETA at this point?
Clearly, we don't get how to have read-only access. Here is our policy (set on user):
{
"Statement": [
{
"Sid": "RContainer",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:HeadObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::my-container",
"arn:aws:s3:::my-container/*"
]
}
]
}
How to get this working?
From to the objectstore-s3 OVH discord channel : Hi, there is a high probability that bucket policies will be available by mid-quarter 2
I confirm that the bucket wide policy does not work but the object one do.
It's a pity this is not clearly stated in the documentation (there is even put-bucket-acl commands)!
If I set a bucket wide policy (as stated in the OVH documentation:
❯ aws s3api put-bucket-acl --bucket my_bucket --acl public-read
The ACL seems to have been correctly set:
❯ aws s3api get-bucket-acl --bucket my_bucket
{
"Owner": {
"DisplayName": "sniped",
"ID": "sniped"
},
"Grants": [
{
"Grantee": {
"Type": "Group",
"URI": "http://acs.amazonaws.com/groups/global/AllUsers"
},
"Permission": "READ"
},
{
"Grantee": {
"DisplayName": "sniped",
"ID": "sniped,
"Type": "CanonicalUser"
},
"Permission": "FULL_CONTROL"
}
]
}
but I cannot access the files in it:
❯ http -h https://my_bucket.s3.gra.io.cloud.ovh.net/derivates/00/1e/95/6e/0066-45a5-8361-00980e89a1aa/sd.jpg
HTTP/1.1 403 Forbidden
If I set the ACL on each file it's ok
❯ aws s3api put-object-acl --bucket my_bucket --key derivates/00/1e/95/6e/0066-45a5-8361-00980e89a1aa/sd.jpg --acl public-read
~
❯ http -h https://mubucket-public.s3.gra.io.cloud.ovh.net/derivates/00/1e/95/6e/0066-45a5-8361-00980e89a1aa/sd.jpg
HTTP/1.1 200 OK
Same thing over here, started a huge migration work to use these new s3 API coming from old swift containers to be faced with this issue, I think it's not explained enough on the docs. I guess we'll have to wait for this to be implemented
# names are redacted for confidentiality reasons
$ curl -I https://MY_BUCKET.s3.gra.io.cloud.ovh.net/payloads/MY_PAYLOAD
HTTP/1.1 403 Forbidden
Content-Type: application/xml
...
Date: Thu, 27 Apr 2023 15:15:15 GMT
when I use object-acl, indeed everything works
$ aws --endpoint-url https://s3.gra.io.cloud.ovh.net s3api put-object-acl --bucket MY_BUCKET --acl public-read --key MY_ARTIFACT --acl public-read
$ curl -I https://MY_BUCKET.s3.gra.io.cloud.ovh.net/payloads/MY_PAYLOAD
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 4306
x-amz-storage-class: STANDARD
....
Date: Thu, 27 Apr 2023 15:18:10 GMT
I think there was a mistake in the ACL doc as it was recently updated. READ at bucket level allows the grantee to only list the objects, you have to set the permission at object level as you did.
We are sorry for the mistake as it was extremely misleading :(
I think there was a mistake in the ACL doc as it was recently updated. READ at bucket level allows the grantee to only list the objects, you have to set the permission at object level as you did.
We are sorry for the mistake as it was extremely misleading :(
Hello, is it possible to apply default ACLs for all new uploaded objects? Thank you!
I think there was a mistake in the ACL doc as it was recently updated. READ at bucket level allows the grantee to only list the objects, you have to set the permission at object level as you did. We are sorry for the mistake as it was extremely misleading :(
Hello, is it possible to apply default ACLs for all new uploaded objects? Thank you!
Hi, you can directly set ACLs on the object when you upload it. Examples:
- if you are using the cli:
aws s3api put-object --bucket my-bucket --key my-object --body path/to/object --ACL public-read - if you are using boto library in Python: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/put_object.html
I'd really love to see that implemented
Do not hesitate to like the issue to help us prioritize ;)
Indeed the current documentation is very misleading, I have been looking for this for hours. It may be interesting to add documentation about this not being supported yet.
I have ~800k objects in my OVH S3 bucket, am I supposed to iterate over each one of them with put-object-acl ? It will take ages...
Edit: a quick fix for this could be having a UI on the manager, you type a prefix (can be empty for root), you a select an available ACLs then apply to all objects.
I have ~800k objects in my OVH S3 bucket, am I supposed to iterate over each one of them with
put-object-acl? It will take ages...Edit: a quick fix for this could be having a UI on the manager, you type a
prefix(can be empty for root), you a select an available ACLs then apply to all objects.
Thanks for the suggestion, we will take that into consideration, stay tuned :)
@lason-ovh any update about that?
It is a simple use case for website:
- readonly for public for all bucket files
- read/write for one user used by the website to do uploads
I guess we have no choice to iterate on all object to set acl or going to AWS to have a complete S3 api.
What's the status of this? Is it already possible to have files / buckets with public access? If so please link me a step by step guide.
I want to host my Podcast Mp3s via OVHs S3.
Thanks a lot.
What's the status of this? Is it already possible to have files / buckets with public access? If so please link me a step by step guide.
I want to host my Podcast Mp3s via OVHs S3.
Thanks a lot.
Hi @pinguin999, first of all, thank you for your commitment with OVHcloud and your interest in this feature. We are currently working very hard to make sure this feature meets the highest quality standards and is delivered very soon.
Back to your use case, currently, it is quite possible to make your bucket "public" by setting the ACLs on your bucket and its objects. You could use the "public-read" predefined ACL as mentioned in our doc.
You will have to set the ACLs for all your objects though which can be a little tedious that's why bucket policies is great because it offers an elegant solution to do that.
In addition to having no effect on the objects from the ACLs bucket, I can't get the ACLs on an object and I cannot put acl on the object when its uploaded from the OVH web manager:
$ aws s3api get-object-acl --bucket myhollybucket --key test1.txt
An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied.
$ aws s3api put-object-acl --bucket myhollybucket --acl public-read --key test1.txt
An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied.
But I can download the object with my user. My user have "FULL_CONTROL" on the bucket. Putting ACL on objects uploaded with awscli works. Do the OVH manager set bad user on files ?
In addition to having no effect on the objects from the ACLs bucket, I can't get the ACLs on an object and I cannot put acl on the object when its uploaded from the OVH web manager:
$ aws s3api get-object-acl --bucket myhollybucket --key test1.txt An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied. $ aws s3api put-object-acl --bucket myhollybucket --acl public-read --key test1.txt An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied.But I can download the object with my user. My user have "FULL_CONTROL" on the bucket. Putting ACL on objects uploaded with awscli works. Do the OVH manager set bad user on files ?
The OVH web manager uses a different user and a different set of credentials than the user you associate with your bucket during creation. Hence, when you upload an object via the UI, its owner is not the same as the user you have configured in your cli. Moreover, the web manager does not offer the option to set the acl on objects, you will have to set them via the cli.
In addition to having no effect on the objects from the ACLs bucket, I can't get the ACLs on an object and I cannot put acl on the object when its uploaded from the OVH web manager:
$ aws s3api get-object-acl --bucket myhollybucket --key test1.txt An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied. $ aws s3api put-object-acl --bucket myhollybucket --acl public-read --key test1.txt An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied.But I can download the object with my user. My user have "FULL_CONTROL" on the bucket. Putting ACL on objects uploaded with awscli works. Do the OVH manager set bad user on files ?
The OVH web manager uses a different user and a different set of credentials than the user you associate with your bucket during creation. Hence, when you upload an object via the UI, its owner is not the same as the user you have configured in your cli. Moreover, the web manager does not offer the option to set the acl on objects, you will have to set them via the cli.
I don't understand, the bucket was created from the OVH manager and my user have FULL CONTROL on the bucket with get-bucket-acl, If different credentials are used, why I can see the bucket ACLs, I can set bucket ACLs, can download / upload objects but can't put acl on objects (becauses objects upload from OVH manager don't inherit the user selected on bucket creation)? So avoid OVH manager.