delta icon indicating copy to clipboard operation
delta copied to clipboard

[Spark] DynamoDBCommitOwner: add logging, get dynamic confs from sparkSession

Open dhruvarya-db opened this issue 9 months ago • 0 comments

Which Delta project/connector is this regarding?

  • [X] Spark
  • [ ] Standalone
  • [ ] Flink
  • [ ] Kernel
  • [ ] Other (fill in here)

Description

Updates DynamoDBCommitOwner:

  • Added logging around table creation flow
  • Get wcu, rcu, and awsCredentialsProvider from SparkSession
  • Return -1 as the table version if registerTable has already been called but no actual commits have gone through the owner. This is done by tracking an extra flag in DynamoDB.

How was this patch tested?

Existing tests

Does this PR introduce any user-facing changes?

Yes, introduces new configs (see DeltaSQLConf changes) which can be used to configure the DynamoDBCommitOwner.

dhruvarya-db avatar May 22 '24 00:05 dhruvarya-db