Accessing bucket with public access to specific prefixes

[sbesson]

I am encountered an issue while trying to mount a bucket where public access has only been configured via policy for certain prefixes. More specifically, I am interested in https://github.com/broadinstitute/cellpainting-gallery.

The root of the bucket is unavailable in an anonymous manner:

$ aws s3 ls s3://cellpainting-gallery/ --no-sign-request

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

but some prefixes are e.g.:

$ aws s3 ls s3://cellpainting-gallery/cpg0012-wawer-bioactivecompoundprofiling/ --no-sign-request
                           PRE broad/

I was hopeful the <bucket:prefix> would allow to use goofys without access key. Unfortunately, this does not seem to be the case with the current release. The following operation fails because of the 403 error code when calling detectBucketLocationByHEAD:

[sbesson@pilot-idrfire-omeroreadwrite ~]$ sudo /opt/goofys -f --debug_s3 -o allow_other cellpainting-gallery:cpg0012-wawer-bioactivecompoundprofiling/ /cellpainting-gallery
2022/09/29 18:51:09.549562 s3.DEBUG HEAD https://s3.amazonaws.com/cellpainting-gallery = 403 [us-east-1]
2022/09/29 18:51:09.605983 s3.DEBUG DEBUG: Sign Request s3/HeadObject failed, not retrying, error NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2022/09/29 18:51:09.606060 s3.ERROR code=NoCredentialProviders msg=no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors, err=<nil>

2022/09/29 18:51:09.606098 main.ERROR Unable to access 'cellpainting-gallery': NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2022/09/29 18:51:09.606121 main.FATAL Mounting file system: Mount: initialization failed

Trying to force the region to skip also fails

[sbesson@pilot-idrfire-omeroreadwrite ~]$ sudo /opt/goofys -f --debug_s3 -o allow_other --region=us-east-1 cellpainting-gallery:cpg0012-wawer-bioactivecompoundprofiling/ /cellpainting-gallery
2022/09/29 18:50:54.710596 s3.DEBUG DEBUG: Sign Request s3/HeadObject failed, not retrying, error NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2022/09/29 18:50:54.710778 s3.ERROR code=NoCredentialProviders msg=no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors, err=<nil>

2022/09/29 18:50:54.710808 main.ERROR Unable to access 'cellpainting-gallery': NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2022/09/29 18:50:54.710828 main.FATAL Mounting file system: Mount: initialization failed

The Init API allows to pass a key which is used for testing the bucket which sounds promising as it is possible to find a key to a public object. Is there a way to configure this API via the command-line utility?

[kahing]

we should make Init use a random key under the prefix