Accommodate separate set of credentials to access DynamoDB

[MusKaya]

Description

When using an S3 compatible storage other than AWS S3, one has a set of access and secret key for the storage and another set for the DynamoDB. In this case, we need an option to provide separate credentials for DynamoDB access.

Use Case Concurrent writes to an S3 compatible storage that does not share the credentials as DynamoDB.

Related Issue(s)

[tustvold]

Which S3 compatible store are you using, most of them shouldn't need DynamoDb as they support conditional writes natively

[MusKaya]

Which S3 compatible store are you using, most of them shouldn't need DynamoDb as they support conditional writes natively

Hi @tustvold, I'm using Linode Object Storage (which has strong consistency).

[tustvold]

So strong consistency is a necessary but insufficient condition here, to not need locking you would need support for conditional writes. I have not been able to ascertain if it supports this.

[MusKaya]

I experimented with AWS_S3_ALLOW_UNSAFE_RENAME='true' and observed that concurrent writes overwrite the json files and the later one wins the race condition.

We need to provide the DynamoDB credentials to enable concurrent writes and the object storage & DynamoDB require separate pairs of secret & access keys.

Can we consider adding optional parameters (under DeltaTable storage_options) to take DynamoDB credentials in case they differ from S3 credentials?