delta
delta copied to clipboard
[Spark] DynamoDBCommitOwner: add logging, get dynamic confs from sparkSession
Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)
Description
Updates DynamoDBCommitOwner:
- Added logging around table creation flow
- Get wcu, rcu, and awsCredentialsProvider from SparkSession
- Return -1 as the table version if registerTable has already been called but no actual commits have gone through the owner. This is done by tracking an extra flag in DynamoDB.
How was this patch tested?
Existing tests
Does this PR introduce any user-facing changes?
Yes, introduces new configs (see DeltaSQLConf changes) which can be used to configure the DynamoDBCommitOwner.