seatunnel
seatunnel copied to clipboard
apache
[Bug] [deploy] Version 2.1.1. No configuration setting found for key 'env'.
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
Version 2.1.1, when using spark on yarn mode to start,报org.apache.seatunnel.shade.com .typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'. Have you ever encountered
SeaTunnel Version
2.1.1
SeaTunnel Config
env {
spark.app.name = "seatunnel_mongo_to_ck_core_device_demo"
# You can set spark configuration here
# see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "2g"
}
source {
mongodb {
readconfig.uri = "mongodb://core_user:[email protected]:27018/gizwits_core"
readconfig.database = "gizwits_core"
readconfig.collection = "device"
readconfig.password = "xxx"
readconfig.spark.mongodb.input.partitioner = "MongoPaginateBySizePartitioner"
schema="{\"created_at\":\"date\",\"updated_at\":\"date\",\"product_key\":\"string\",\"did\":\"string\", \"passcode\":\"string\", \"mac\":\"string\"}"
result_table_name = "core_device"
}
}
transform {
sql {
sql = "SELECT created_at, updated_at, product_key,did,passcode,mac, '2022-05-10' as dt FROM core_device"
}
json {
source_field = "created_at"
new_type = "datetime"
}
json {
source_field = "updated_at"
new_type = "datetime"
}
json {
source_field = "dt"
new_type = "date"
}
}
sink {
clickhouse {
host = "xxx:8123"
database = "test"
table = "ods_core_device"
fields = ["created_at", "updated_at", "product_key", "did", "passcode", "mac", "dt"]
username = "default"
}
}
Running Command
./bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config ./config/to_clickhouse_conf/mongo_2_ck_core_device_demo.conf.template
Error Exception
Version 2.1.1, when using spark on yarn mode to start,[报org.apache.seatunnel.shade.com](http://xn--org-pb6f.apache.seatunnel.shade.com/) .typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'. Have you ever encountered
22/05/11 11:37:03 ERROR Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:44)
at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:36)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
22/05/11 11:37:03 ERROR Seatunnel:
===============================================================================
Flink or Spark Version
spark 2.4.3
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
@BenJFan Here is the full log. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/emr/yarn/local/usercache/root/filecache/162/__spark_libs__8798545789667083542.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/service/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 22/05/11 11:37:03 ERROR Seatunnel:
===============================================================================
22/05/11 11:37:03 ERROR Seatunnel: Fatal Error,
22/05/11 11:37:03 ERROR Seatunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
22/05/11 11:37:03 ERROR Seatunnel: Reason:No configuration setting found for key 'env'
22/05/11 11:37:03 ERROR Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
at org.apache.seatunnel.config.ExecutionContext.
22/05/11 11:37:03 ERROR Seatunnel:
22/05/11 11:37:03 ERROR ApplicationMaster: User class threw exception: org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
at org.apache.seatunnel.config.ExecutionContext.
@BenJFan Can you help me to check this? thanks ！
I will check this, please wait a mement.
flink run -t yarn-per-job -c org.apache.seatunnel.core.flink.SeatunnelFlink /usr/local/apache-seatunnel-incubating-2.1.2/lib/seatunnel-core-flink.jar --config ./config/flink.streaming.conf.template
flink yarn-application-mode also meet the same problem,this may be bug, because it's ok when i run on yarn-per-job mode
I can't reproduce this error. Ran fine on my env. Maybe already fix it on dev when 2.1.2 not release.
I can't reproduce this error. Ran fine on my env. Maybe already fix it on dev when 2.1.2 not release.
Ok, I will close this issue. If the bug reproduce in 2.1.3/dev/2.2.0 again, please reopen this issue.
Search before asking
- [x] I had searched in the issues and found no similar issues.
What happened
Version 2.1.1, when using spark on yarn mode to start,报org.apache.seatunnel.shade.com .typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'. Have you ever encountered
SeaTunnel Version
2.1.1
SeaTunnel Config
env { spark.app.name = "seatunnel_mongo_to_ck_core_device_demo" # You can set spark configuration here # see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties spark.executor.instances = 2 spark.executor.cores = 1 spark.executor.memory = "2g" } source { mongodb { readconfig.uri = "mongodb://core_user:[email protected]:27018/gizwits_core" readconfig.database = "gizwits_core" readconfig.collection = "device" readconfig.password = "xxx" readconfig.spark.mongodb.input.partitioner = "MongoPaginateBySizePartitioner" schema="{\"created_at\":\"date\",\"updated_at\":\"date\",\"product_key\":\"string\",\"did\":\"string\", \"passcode\":\"string\", \"mac\":\"string\"}" result_table_name = "core_device" } } transform { sql { sql = "SELECT created_at, updated_at, product_key,did,passcode,mac, '2022-05-10' as dt FROM core_device" } json { source_field = "created_at" new_type = "datetime" } json { source_field = "updated_at" new_type = "datetime" } json { source_field = "dt" new_type = "date" } } sink { clickhouse { host = "xxx:8123" database = "test" table = "ods_core_device" fields = ["created_at", "updated_at", "product_key", "did", "passcode", "mac", "dt"] username = "default" } }Running Command
./bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config ./config/to_clickhouse_conf/mongo_2_ck_core_device_demo.conf.templateError Exception
Version 2.1.1, when using spark on yarn mode to start,[报org.apache.seatunnel.shade.com](http://xn--org-pb6f.apache.seatunnel.shade.com/) .typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'. Have you ever encountered 22/05/11 11:37:03 ERROR Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env' at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41) at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47) at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49) at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:44) at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:36) at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48) at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684) 22/05/11 11:37:03 ERROR Seatunnel: ===============================================================================Flink or Spark Version
spark 2.4.3
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
hi is this problom solved in the newest dev branch? I met this again..
