[SUPPORT] [BUG] SparkSQL Insert Into other catalog Error

[Kyofin]

trafficstars

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at [email protected].

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

I set a catalog named mysql using JdbcCatalog In spark , when i insert into mysql.tb1 , some error happen because hudi just like below. The hudi.SparkAdpter will affect my spark-default.conf is :

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version : 0.11

• Spark version : 3.2.1

• Hive version : 2.3.5

• Hadoop version : 2.7.3

• Storage (HDFS/S3/GCS..) : HDFS

• Running on Docker? (yes/no) : no

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[Kyofin]

I use iceberg hadoop catalog named local and hudi using spark_catalog at same spark driver ,same error will happen

[yihua]

@xiarixiaoyao @YannByron @XuQianJin-Stars @alexeykudinkin any of you can help here?

[YannByron]

@huzk8 please give the whole stack trace to help me address this. and maybe you should use the catalog.database.table format to insert into, not mysql.st which only have catalog and table i think.

[YannByron]

update: @yihua @huzk8 i can reproduce your case that only use catalog.table:

but if use catalog.database.table, it will raise this error that need to be resolved by this ticket: https://issues.apache.org/jira/browse/HUDI-4503.

[codope]

@huzk8 Can you please try the patch authored by @YannByron #6264

[leesf]

update: @yihua @huzk8 i can reproduce your case that only use catalog.table:

but if use catalog.database.table, it will raise this error that need to be resolved by this ticket: https://issues.apache.org/jira/browse/HUDI-4503.

this is a warn log and I think insert into other catalog using three parts(catalogName.dbName.tableName) should work fine with the patch(https://github.com/apache/hudi/pull/5957)

[nsivabalan]

@huzk8 : do you have any updates on this regard

[alexeykudinkin]

@nsivabalan this will be addressed by https://github.com/apache/hudi/pull/6264

[nsivabalan]

@huzk8 : can you give https://github.com/apache/hudi/pull/6264 and let us know if the issue is resolved.

[alexeykudinkin]

@nsivabalan let's close this one. We have HUDI-4503 to track.