Fetching policy bundle from AWS-S3 fails due to missing header

[kbalthaser]

Describe the bug When the opal-server is configured to load a policy bundle remotely ( OPAL_POLICY_BUNDLE_SERVER_TYPE=AWS-S3), it signs the request using AWS Signature V4.

This header set is missing the required header x-amz-content-sha256. The request will fail.

Per https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html :

The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there is no payload, you must provide the hash of an empty string.

--
To Reproduce Configure opal server with ENV variables required to fetch from S3:

• OPAL_POLICY_BUNDLE_SERVER_TOKEN_ID=someuser
• OPAL_POLICY_SOURCE_TYPE=API
• OPAL_POLICY_BUNDLE_SERVER_TYPE=AWS-S3
• OPAL_POLICY_BUNDLE_URL=https://somebucket.s3.amazonaws.com
• OPAL_POLICY_BUNDLE_SERVER_TOKEN=somesecret

The server will make a request, lacking the x-amz-content-sha256 header. The request will error out, with s3 returning an XML error message.

Expected behavior The opal-server should be adding the header x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 to the request. This is the checksum of an empty string, which is required for our read operation.

OPAL version

• Version: [head]

[orweis]

HI @kbalthaser - thanks for reporting! This feature in OPAL was a great community contribution (#473 by @cbat98 💜) - feel like maybe contributing a PR yourself? If yes, this would be the place to start - https://github.com/permitio/opal/blob/fc23168de2573671063e3cb280284681f63ed04e/packages/opal-common/opal_common/sources/api_policy_source.py#L125

[yemaney]

Any reason the pull request for this didn't get merged?

Got errors in opal-server logs which I believe is the same thing.

Unless I'm missing something, I'd be surprised if this S3 feature has actually works for anyone. Since the AWS Docs are pretty explicit about the x-amz-content-sha256 being required in the headers.

Their example Example: GET Object request.

GET /test.txt HTTP/1.1
Host: examplebucket.s3.amazonaws.com
Authorization: SignatureToBeCalculated
Range: bytes=0-9 
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20130524T000000Z 

Should mean the auth header return dict should look like this at the minimum.

def build_aws_rest_auth_headers():
   ...
  return {
          "x-amz-date": amzdate,
          "Authorization": authorization_header,
          "x-amz-content-sha256": payload_hash # hashlib.sha256("".encode("utf-8")).hexdigest()
  }

opal server Logs

2024-03-16T02:00:27.575935+0000 | opal_common.sources.api_policy_source   |WARNING | server connection error: 
ContentTypeError(RequestInfo(url=URL('https://bucket.s3.amazonaws.com/folder/subfolder/bundle.tar.gz'), method='GET', 
headers=<CIMultiDictProxy('Host': 'bucket.s3.amazonaws.com', 'content-type': 'application/gzip', 'x-amz-date': 
'20240316T020027Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=AKXXX/20240316/us-east-1/s3/aws4_request, 
SignedHeaders=host;x-amz-date, Signature=5b56XXX', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 
'Python/3.10 aiohttp/3.8.5')>, real_url=URL('https://bucket.s3.amazonaws.com/folder/subfolder/bundle.tar.gz')), (), 
message='Attempt to decode JSON with unexpected mimetype: application/xml', headers=<CIMultiDictProxy('x-amz-request-
id': '01XBQPFXXX', 'x-amz-id-2': '85y92urXziD3XXXo=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 
'Sat, 16 Mar 2024 02:00:27 GMT', 'Server': 'AmazonS3', 'Connection': 'close')>)

[roekatz]

@yemaney Sorry for delaying this. I've reviewed that PR - @kbalthaser let us know if you can fix my few comments - and we'll get this merged.