delta icon indicating copy to clipboard operation
delta copied to clipboard
verified publisher icon delta-io

always include CatalogTable#storage#properties into data source options

Open cloud-fan opened this issue 1 year ago • 1 comments

Which Delta project/connector is this regarding?

  • [x] Spark
  • [ ] Standalone
  • [ ] Flink
  • [ ] Kernel
  • [ ] Other (fill in here)

Description

DeltaTableV2 uses the options to create DeltaLog, assuming that the options should always be consistent with the one in CatalogTable#storage#properties. This may not be true as custom catalogs may add more table storage properties on the fly, like Unity Catalog.

How was this patch tested?

Tested locally with UC. It's not an issue for HMS.

Does this PR introduce any user-facing changes?

no

cloud-fan avatar Aug 13 '24 06:08 cloud-fan

To avoid risks, I've made this PR more surgical: only extract file system options from the table storage properties and only apply them when they are not defined in the original options. This is the smallest change we need to make for UC.

cc @prakharjain09 @scovich @tdas

cloud-fan avatar Aug 21 '24 15:08 cloud-fan

For example, given a conflict between options and a spark conf, I believe the former prevails?

Yes, this is to maximize the backward compatibility.

cloud-fan avatar Aug 27 '24 05:08 cloud-fan