seatunnel
seatunnel copied to clipboard
Seatunnel-2.1.3 How to specify a spark file?
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
Seatunnel-2.1.3 How to specify a spark file? Spark2.4 is used. The hive table cannot be found. The hive site is specified Xml is normal.
SeaTunnel Version
Seatunnel-2.1.3
SeaTunnel Config
cat /tmp/mysql_tt3.sh
env {
spark.app.name = "ads_product_sale_d"
spark.executor.instances = 1
spark.dynamicAllocation.maxExecutors = 20
spark.executor.cores = 1
spark.executor.memory = "8g"
spark.sql.catalogImplementation = "hive"
spark.sql.hive.verifyPartitionPath = "true"
}
source {
hive {
pre_sql = "select * from mid.player"
result_table_name = "player_hive"
}
}
transform {
}
sink {
Doris {
fenodes = "172.17.209.4:8030"
database = "example_db"
table = "player_starrocks"
user = "root"
password = "123456"
batch_size=10000
doris.column_separator="\t"
doris.columns="id,name,age"
}
}
Running Command
su hive -s /bin/bash -c "/app/seatunnel-2.1.3/bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config /tmp/mysql_tt3.sh"
Error Exception
Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: `mid`.`player_parquet`; line 1 pos 14;
Flink or Spark Version
spark-2.4
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Do you want some other people help you fix it?
Yes, can I help you? thank you
Do you want some other people help you fix it?
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.