Unable to upload to S3 bucket owned by another AWS account even with write permissions

[sarahwooders]

I have a case where AWS account A has read permissions to S3 bucket A, and is trying to write to S3 bucket B owned by AWS account B with read/write permission to A. The bucket policy says the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "...",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::B"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::bucket",
                "arn:aws:s3:::bucket/*"
            ]
        }
    ]
}

Running skyplane cp -r s3://bucket-A/ s3://bucket-B/ throws the error

❌ AWSServer(region_tag=aws:us-east-1, instance_id=i-0c35275fe4ef69016) encountered error:
Traceback (most recent call last):
  File "/pkg/skyplane/gateway/gateway_obj_store.py", line 95, in worker_loop
    retry_backoff(
  File "/pkg/skyplane/utils/retry.py", line 30, in retry_backoff
    raise e
  File "/pkg/skyplane/utils/retry.py", line 27, in retry_backoff
    return fn()
  File "/pkg/skyplane/utils/imports.py", line 33, in wrapped
    return fn(*modules_imported, *args, **kwargs)
  File "/pkg/skyplane/obj_store/s3_interface.py", line 167, in upload_object
    s3_client.upload_part(
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 938, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied

The AWS CLI command aws cp --recursive s3://bucket-A/ s3://bucket-B/ works fine (authenticated with account A).

I looked into it a bit and it seems like multipart uploads require KMS permissions https://github.com/aws/aws-cli/issues/4251. This might be a common use-case worth looking into.