hudi icon indicating copy to clipboard operation
hudi copied to clipboard

[MINOR] if sync mode is glue, fix the sync tool class

Open prabodh1194 opened this issue 7 months ago • 6 comments

Change Logs

While writing the Hudi table in spark, I want to sync the table to Glue.

However, the sync mode: "hoodie.datasource.hive_sync.mode": "glue" is actually not applicable in spark.

It instead works using this setting: "hoodie.meta.sync.client.tool.class": "org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool".

I believe that the former config offers a better developer experience and is rather cheap to implement. Raising the PR for same as well.

Impact

Some spark code might spurious hoodie option of "hoodie.datasource.hive_sync.mode": "glue" but is actually writing to some other hive metastore. The behavior might start breaking.

Risk level (write none, low medium or high below)

na

Documentation Update

"none"

Contributor's checklist

  • [x] Read through contributor's guide
  • [x] Change Logs and Impact were stated clearly
  • [ ] Adequate tests were added if applicable
  • [ ] CI passed

prabodh1194 avatar Jun 29 '24 18:06 prabodh1194