Option to specify SSE-KMS or SSE-S3 encryption when writing data with load_catalog / append

[nagoorga-create]

Question

Hello team,

I am using pyiceberg to load data into an Iceberg table stored in Amazon S3. While doing this, I am facing an explicit deny from an AWS Service Control Policy (SCP) that blocks multipart uploads without encryption. I cannot modify the SCP.

Error excerpt: OSError: When initiating multiple part upload for key 'iceberg/DEV/dataset/test_4_matdoc/metadata/...' in bucket 'pt-s3-project-bucketname': AWS Error ACCESS_DENIED during CreateMultipartUpload operation: User: arn:aws:sts::... is not authorized to perform: s3:PutObject with an explicit deny in a service control policy This happens during calls like:

def load_iceberg_table(table, arrow_table): catalog = load_catalog("glue", **{"type": "glue"}) iceberg_table: Table = catalog.load_table(f"{DATABASE}.{table}") try: logger.info("Appending data to Iceberg table...") iceberg_table.append(df=arrow_table) logger.info("Successfully appended data to Iceberg table.") except ClientError as e: logger.error(f"Iceberg append ClientError: {e}") raise except Exception as e: logger.error(f"Unexpected Iceberg error: {e}") raise

From my understanding, pyiceberg uses S3 multipart upload under the hood, but I haven’t found a documented way to configure SSE-KMS or SSE-S3 parameters for these writes.

Question: Is there currently a way to pass S3 upload parameters (like ServerSideEncryption, SSEKMSKeyId) via load_catalog, append, or FileIO configuration? If not, could this be added as a feature so that environments with encryption-required SCPs can still use pyiceberg without policy changes?

Thanks!

[aarslanargin]

Hi, I’m running into a similar but slightly different problem when writing to an S3 bucket with PyIceberg using SSE-KMS.

Environment:

• PyIceberg version: latest
• Catalog: GlueCatalog
• Bucket encryption: aws:kms with enforced KMS key
• Context: ECS task in a VPC

Error:

AWS Error ACCESS_DENIED during CreateMultipartUpload operation: 
User: arn:aws:sts::<account_id>:assumed-role/<ecs-role> is not authorized to perform: s3:PutObject 
on resource: "arn:aws:s3:::<bucket>/iceberg/.../metadata/00000-<uuid>.metadata.json" 
with an explicit deny in a resource-based policy

Observations:

Removing all KMS references (bucket policy) allows the write to succeed.

Using S3FileSystem / s3fs in Python works correctly with the same KMS key.

It seems PyIceberg does not send the required SSE-KMS headers during multipart upload, causing the access denial.

I’m sharing this in case it helps identify whether multipart uploads with KMS are fully supported or if this is a gap in the current implementation.