hudi icon indicating copy to clipboard operation
hudi copied to clipboard

Published 20 hours ago • verified publisher icon apache

[SUPPORT] hudi-examples-dbt not running with spark thrift server

Open sambhav13 opened this issue 2 years ago • 4 comments

Tips before filing an issue

  • Have you gone through our FAQs?

  • Join the mailing list to engage in conversations and get faster support at [email protected].

  • If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced Tried Running https://github.com/apache/hudi/tree/master/hudi-examples/hudi-examples-dbt . Started the spark Thrift server on 10000 port and configured that in dbt. On running dbt run , error comes saying org.apache.hudi.exception.HoodieException: 'hoodie.table.name' must be set.

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

  1. Start thrift server locally using ./sbin/start-thriftserver.sh --jars hudi-spark-bundle_2.11-0.10.1.jar using spark-3.1.1-bin-hadoop3.2
  2. Set up ~/.dbt/profiles.yaml learnDbt: outputs: dev: host: localhost method: thrift port: 10000 schema: sales threads: 1 type: spark target: dev

spark: outputs: dev: host: localhost method: thrift port: 10000 schema: sales threads: 1 type: spark target: dev

  1. dbt debug to test connectivity
  2. dbt run to create models.

Expected behavior

As expected on the dbt run command the command should be able to successfully create models.

Environment Description local cloned repo on macOs.

  • Hudi version : 0.11

  • Spark version : 3.1.1

  • Hive version :

  • Hadoop version :

  • 3.2

  • Storage (HDFS/S3/GCS..) : Local Storage

  • Running on Docker? (yes/no) : no

Additional context

Add any other context about the problem here.

Stacktrace " Database Error 10:38:28 org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.hudi.exception.HoodieException: 'hoodie.table.name' must be set" Add the stacktrace of the error. dbt-run-with_hudi-examples-dbt

sambhav13 avatar Jul 17 '22 19:07 sambhav13

hey @vingov can you take a look please?

xushiyan avatar Jul 18 '22 00:07 xushiyan

@sambhav13 Does it work when using spark-submit/sql?

minihippo avatar Jul 18 '22 12:07 minihippo

@minihippo Can you elaborate a bit what do you mean by spark-submit/sql? Can you cite a small example what would you like me try? What I did was just clone the hudi repo and tried with dbt with spark thrift server configured as backend engine for dbt to run the hudi-examples-dbt models. I think it should not matter what way spark is being run , It could be due to some property missing but I am relying on the hudi-examples-dbt to have all that already .

sambhav13 avatar Jul 18 '22 14:07 sambhav13

@vingov : can u assist here please. @minihippo : if you have time, can you follow up as well

nsivabalan avatar Aug 09 '22 21:08 nsivabalan

@sambhav13 We confirmed it works well with spark thrift server. Tried hudi 0.12.3 , 0.13.1 and 0.14.0 and all works good.

Sorry for the delay in triaging this. Do you still face this issue. I can help if you still not able to make it run. Thanks.

ad1happy2go avatar Nov 16 '23 19:11 ad1happy2go

@sambhav13 I'm updating the instructions in the dbt example (using spark 3.2 and hudi 0.14.0). Please check this out and let us know if it helps.

https://github.com/apache/hudi/blob/af912a3fed7270708fad935b7df55fb508cd5536/hudi-examples/hudi-examples-dbt/README.md

xushiyan avatar Nov 22 '23 07:11 xushiyan

closing as solution provided

xushiyan avatar Nov 27 '23 16:11 xushiyan