seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

Seatunnel-2.1.3 How to specify a spark file?

Open 2416210017 opened this issue 2 years ago • 3 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

Seatunnel-2.1.3 How to specify a spark file? Spark2.4 is used. The hive table cannot be found. The hive site is specified Xml is normal. image

image Uploading image.png…

SeaTunnel Version

Seatunnel-2.1.3

SeaTunnel Config

cat /tmp/mysql_tt3.sh

env {
   spark.app.name = "ads_product_sale_d"
   spark.executor.instances = 1
   spark.dynamicAllocation.maxExecutors = 20
   spark.executor.cores = 1
   spark.executor.memory = "8g"
   spark.sql.catalogImplementation = "hive"
   spark.sql.hive.verifyPartitionPath = "true"
  }
  
source  {

       hive {
        pre_sql = "select * from mid.player"
        result_table_name = "player_hive"
       }
	   
  }
  
transform {
}

  sink {
      Doris {
	       fenodes = "172.17.209.4:8030"
          database = "example_db"
          table = "player_starrocks"
          user = "root"
          password = "123456"
          batch_size=10000
          doris.column_separator="\t"
          doris.columns="id,name,age"
      }
          
  }

Running Command

su hive -s /bin/bash -c "/app/seatunnel-2.1.3/bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config /tmp/mysql_tt3.sh"

Error Exception

Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: `mid`.`player_parquet`; line 1 pos 14;

Flink or Spark Version

spark-2.4

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

2416210017 avatar Sep 28 '22 03:09 2416210017

Do you want some other people help you fix it?

EricJoy2048 avatar Sep 28 '22 06:09 EricJoy2048

Yes, can I help you? thank you

Do you want some other people help you fix it?

2416210017 avatar Sep 28 '22 06:09 2416210017

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] avatar Oct 29 '22 00:10 github-actions[bot]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

github-actions[bot] avatar Nov 05 '22 00:11 github-actions[bot]