Hongyue/Steve Zhang

Spark: Support Bulk deletion in expire-snapshots if fileIO allows

1

In #4052, S3 fileIO now implement a new interfaces to support S3 batch deletion, this PR introduce it for Spark expire-snapshots procedure to conditionally support delete files in batch if...

Missing dynamodb permission on recommended IAM user policy

2

Unable to create cloudformation stack "Data Service" due to `User: arn:aws:iam::$AWSACCOUNTID:user/blox is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-2:388892025991:table/Environments` The recommended IAM user policy on dev branch [README.md](https://github.com/blox/blox/blob/dev/README.md) used...

Core: Add data sequence number as derived column to files metadata table

1

This PR add data_sequence_number as derived/virtual column on all files metadata table, enables query like ``` SELECT data_sequence_number FROM iceberg.db.table.files ``` without change the avro schema for files.

Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction

4

Goal: Attempt to clean up the dangling deletes as part of Spark RewriteDataFilesAction, it can be controlled by feature flag `remove-dangling-deletes` and it's by default turned on. Most of the...

Spark: support read of partition metadata column when table is over 1k

1

support of `SELECT *, _partition from iceberg.foo.bar` when foo.bar table has over 1000 columns defined