seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard
verified publisher icon apache

ST Parse Json configuration file has some issue

Open chenhu opened this issue 3 years ago • 2 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

ST Parse Json configuration file has some issue, I has a json configuration file,and the parse result is :

2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 22/09/28 14:19:09 INFO ConfigBuilder: Loading config file: /tmp/datax-executor/jsonjobTmp-abf3e9ce6c6046bf8377598857eba6ef.conf 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 22/09/28 14:19:09 INFO ConfigBuilder: parsed config file: { 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "transform" : [ 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] [ 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders_t", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "Sql", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "select toYYYYMMDD(createdate) as day, * from orders" 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ] 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ], 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "sink" : [ 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] [ 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "system", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "com.mysql.jdbc.driver", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "Jdbc", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "root", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "jdbc:mysql://192.168.110.228:3306/parking", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "true" 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ] 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ], 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "source" : [ 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] [ 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "system", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "1", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "com.mysql.jdbc.Driver", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "ID", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "20", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "10", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "Jdbc", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "root", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "orders", 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "jdbc:mysql://192.168.110.228:3306/parking?useSSL=true&Unicode=true&characterEncoding=UTF-8" 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ] 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] ], 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "env" : { 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] "spark.default.parallelism" : "10" 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] } 2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] }

SeaTunnel Version

2.1.3

SeaTunnel Config

{
    "env": {
        "spark.default.parallelism": "10"
    },
    "source": [
    {
        "password": "system",
        "jdbc.lowBound": "1",
        "driver": "com.mysql.jdbc.Driver",
        "jdbc.partitionColumn": "ID",
        "jdbc.upperBound": "20",
        "jdbc.numberPartitions": "10",
        "result_table_name": "orders",
        "plugin_name": "Jdbc",
        "user": "root",
        "table": "orders",
        "url": "jdbc:mysql://192.168.110.228:3306/parking?useSSL=true&Unicode=true&characterEncoding=UTF-8"
    }
    ],
    "transform": [
    {
        "result_table_name": "orders_t",
        "source_table_name": "orders",
        "plugin_name": "Sql",
        "sql": "select  toYYYYMMDD(createdate) as  day, * from orders"
    }
    ],
     "sink": [
     {
        "password": "system",
        "driver": "com.mysql.jdbc.driver",
        "dbTable": "orders",
        "source_table_name": "orders",
        "plugin_name": "Jdbc",
        "user": "root",
        "url": "jdbc:mysql://192.168.110.228:3306/parking",
        "useSsl": "true"
    }]
}

Running Command

./bin/start-seatunnel-spark.sh --master local -c ~/Downloads/test.conf --deploy-mode client

Error Exception

ERROR Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$WrongType: /tmp/datax-executor/jsonjobTmp-abf3e9ce6c6046bf8377598857eba6ef.conf: 1: source has type list of LIST rather than list of OBJECT
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getHomogeneousWrappedList(SimpleConfig.java:452)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObjectList(SimpleConfig.java:460)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfigList(SimpleConfig.java:465)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.config.EnvironmentFactory.checkIsContainHive(EnvironmentFactory.java:62)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:44)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:36)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at java.lang.reflect.Method.invoke(Method.java:498)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53]
2022-09-28 14:20:39 [AnalysisStatistics.analysisStatisticsLog-53] 22/09/28 14:19:10 ERROR Seatunnel:

Flink or Spark Version

spark3.3 on hadoop3

Java or Scala Version

java8

Screenshots

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

chenhu avatar Sep 28 '22 10:09 chenhu

linked #2933

ashulin avatar Sep 30 '22 11:09 ashulin

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] avatar Oct 31 '22 00:10 github-actions[bot]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

github-actions[bot] avatar Nov 08 '22 00:11 github-actions[bot]