seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

Published 20 hours ago • verified publisher icon apache

SqlServer-CDC 没有数据

Open cs3163077 opened this issue 1 year ago • 0 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

SqlServer-CDC 没有数据,也没报错

SeaTunnel Version

2.3.2

SeaTunnel Config

env {
  job.mode = "STREAMING"
}
source {
  SqlServer-CDC {
    result_table_name = "area"    
	username = "**"
	password = "****"
	database-names = ["BI"]
	table-names = ["BI.dbo.area"]
    base-url="jdbc:sqlserver://******:**;databaseName=BI"
  }
}
transform {
}
sink {
     Console {
    }
}

Running Command

./start-seatunnel-spark-2-connector-v2.sh --config /home/apache-seatunnel-incubating-2.3.1/config/cdc.conf

Error Exception

23/06/30 15:24:25 INFO spark.SparkContext: Added JAR file:/home/apache-seatunnel-incubating-2.3.1/starter/seatunnel-spark-2-starter.jar at spark://hadoop3:37437/jars/seatunnel-spark-2-starter.jar with timestamp 1688109865948
23/06/30 15:24:26 INFO executor.Executor: Starting executor ID driver on host localhost
23/06/30 15:24:26 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41040.
23/06/30 15:24:26 INFO netty.NettyBlockTransferService: Server created on hadoop3:41040
23/06/30 15:24:26 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/06/30 15:24:26 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop3:41040 with 366.3 MB RAM, BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO storage.BlockManager: external shuffle service port = 7337
23/06/30 15:24:26 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, hadoop3, 41040, None)
23/06/30 15:24:26 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@41b1f51e{/metrics/json,null,AVAILABLE,@Spark}
23/06/30 15:24:27 INFO scheduler.EventLoggingListener: Logging events to hdfs://hadoop1:8020/user/spark/applicationHistory/local-1688109866035
23/06/30 15:24:27 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
23/06/30 15:24:27 INFO util.Utils: Extension com.cloudera.spark.lineage.NavigatorAppListener not being initialized.
23/06/30 15:24:27 INFO logging.DriverLogger$DfsAsyncWriter: Started driver log file sync to: /user/spark/driverLogs/local-1688109866035_driver.log
23/06/30 15:24:28 INFO discovery.AbstractPluginDiscovery: Load SeaTunnelSource Plugin from /home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
23/06/30 15:24:28 INFO discovery.AbstractPluginDiscovery: Discovery plugin jar: SqlServer-CDC at: file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-cdc-sqlserver-2.3.2.jar
23/06/30 15:24:28 INFO discovery.AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='SqlServer-CDC'} from classpath
23/06/30 15:24:28 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:28 INFO utils.TableDiscoveryUtils: Read list of available databases
23/06/30 15:24:29 INFO utils.TableDiscoveryUtils: 	 list of available databases is: [master, tempdb, model, msdb, BI, BIDEMO, SYSD_BI_DEMO, SYSD_BI_DEV_JAVA, *****, SY_BI_COST, SY_BI_1141, TEST1, SYSD_BI_TEST_20211026, SYSD_BI_DEV_JAVA1, test123, testdatabig, test12134, SYBI_Test_TW, zhuhm_BIDW, bigdatatest, zhuhm_BIDW_E, zhuhm_BIDW1, *****, test_yx, S*****Test, SY_BI_CNCEC, BIDW5, *****, S*****, SY_BI_LSC]
23/06/30 15:24:29 INFO utils.TableDiscoveryUtils: Read list of available tables in each database
23/06/30 15:24:29 INFO utils.TableDiscoveryUtils: 	 including 'BI.dbo.area' for further processing
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: 	 skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线，无法打开该数据库。
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: 	 skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线，无法打开该数据库。
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: 	 skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线，无法打开该数据库。
23/06/30 15:24:29 WARN utils.TableDiscoveryUtils: 	 skipping database 'S*****' due to error reading tables: 由于数据库 'S*****' 离线，无法打开该数据库。
23/06/30 15:24:29 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:29 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:29 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:29 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:29 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:30 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:30 INFO execution.SparkRuntimeEnvironment: register plugins :[file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-cdc-sqlserver-2.3.2.jar]
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Load SeaTunnelTransform Plugin from /home/apache-seatunnel-incubating-2.3.1/lib
23/06/30 15:24:30 INFO execution.SparkRuntimeEnvironment: register plugins :[]
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Load SeaTunnelSink Plugin from /home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Discovery plugin jar: Console at: file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-console-2.3.1.jar
23/06/30 15:24:30 INFO discovery.AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Console'} from classpath
23/06/30 15:24:30 INFO execution.SparkRuntimeEnvironment: register plugins :[file:/home/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-console-2.3.1.jar]
23/06/30 15:24:30 INFO internal.SharedState: loading hive config file: file:/etc/hive/conf.cloudera.hive/hive-site.xml
23/06/30 15:24:30 INFO internal.SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
23/06/30 15:24:30 INFO internal.SharedState: Warehouse path is '/user/hive/warehouse'.
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4966bab1{/SQL,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7f0f84d4{/SQL/json,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@590d6c76{/SQL/execution,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@25791d40{/SQL/execution/json,null,AVAILABLE,@Spark}
23/06/30 15:24:30 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a05dd30{/static/sql,null,AVAILABLE,@Spark}
23/06/30 15:24:31 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
23/06/30 15:24:31 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
23/06/30 15:24:31 INFO util.Utils: Extension com.cloudera.spark.lineage.NavigatorQueryListener not being initialized.
23/06/30 15:24:34 INFO v2.DataSourceV2Strategy: 
Pushing operators to class org.apache.seatunnel.translation.spark.source.SeaTunnelSourceSupport
Pushed Filters: 
Post-Scan Filters: 
Output: Areaid#0, Areaname#1
         
23/06/30 15:24:34 INFO codegen.CodeGenerator: Code generated in 199.1582 ms
23/06/30 15:24:34 INFO v2.WriteToDataSourceV2Exec: Start processing data source writer: org.apache.seatunnel.translation.spark.sink.writer.SparkDataSourceWriter@7b8d6c66. The input RDD has 1 partitions.
23/06/30 15:24:34 INFO spark.SparkContext: Starting job: save at SinkExecuteProcessor.java:123
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Got job 0 (save at SinkExecuteProcessor.java:123) with 1 output partitions
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (save at SinkExecuteProcessor.java:123)
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Parents of final stage: List()
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Missing parents: List()
23/06/30 15:24:34 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at save at SinkExecuteProcessor.java:123), which has no missing parents
23/06/30 15:24:34 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.9 KB, free 366.3 MB)
23/06/30 15:24:35 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.1 KB, free 366.3 MB)
23/06/30 15:24:35 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop3:41040 (size: 3.1 KB, free: 366.3 MB)
23/06/30 15:24:35 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1164
23/06/30 15:24:35 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at save at SinkExecuteProcessor.java:123) (first 15 tasks are for partitions Vector(0))
23/06/30 15:24:35 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
23/06/30 15:24:35 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 16026 bytes)
23/06/30 15:24:35 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
23/06/30 15:24:35 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/connector-cdc-sqlserver-2.3.2.jar with timestamp 1688109865947
23/06/30 15:24:35 INFO client.TransportClientFactory: Successfully created connection to hadoop3/192.168.0.213:37437 after 75 ms (0 ms spent in bootstraps)
23/06/30 15:24:35 INFO util.Utils: Fetching spark://hadoop3:37437/jars/connector-cdc-sqlserver-2.3.2.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp8739054726675673564.tmp
23/06/30 15:24:35 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/connector-cdc-sqlserver-2.3.2.jar to class loader
23/06/30 15:24:35 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/seatunnel-spark-2-starter.jar with timestamp 1688109865948
23/06/30 15:24:35 INFO util.Utils: Fetching spark://hadoop3:37437/jars/seatunnel-spark-2-starter.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp4417905759660334726.tmp
23/06/30 15:24:36 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/seatunnel-spark-2-starter.jar to class loader
23/06/30 15:24:36 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/connector-console-2.3.1.jar with timestamp 1688109865947
23/06/30 15:24:36 INFO util.Utils: Fetching spark://hadoop3:37437/jars/connector-console-2.3.1.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp8783962742653812302.tmp
23/06/30 15:24:36 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/connector-console-2.3.1.jar to class loader
23/06/30 15:24:36 INFO executor.Executor: Fetching spark://hadoop3:37437/jars/seatunnel-transforms-v2.jar with timestamp 1688109865945
23/06/30 15:24:36 INFO util.Utils: Fetching spark://hadoop3:37437/jars/seatunnel-transforms-v2.jar to /tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/fetchFileTemp6075022225571179099.tmp
23/06/30 15:24:36 INFO executor.Executor: Adding file:/tmp/spark-3382b920-11e6-4daf-ad0c-a523df4ead04/userFiles-a4ca48a0-10f3-468f-b5e3-b953bfa97755/seatunnel-transforms-v2.jar to class loader
23/06/30 15:24:36 INFO sink.ConsoleSinkWriter: output rowType: Areaid<STRING>, Areaname<STRING>
23/06/30 15:24:36 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: Read list of available databases
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: 	 list of available databases is: [BI]
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: Read list of available tables in each database
23/06/30 15:24:36 INFO utils.TableDiscoveryUtils: 	 including 'BI.dbo.area' for further processing
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: 	 skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线，无法打开该数据库。
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: 	 skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线，无法打开该数据库。
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: 	 skipping database '*****' due to error reading tables: 由于数据库 '*****' 离线，无法打开该数据库。
23/06/30 15:24:36 WARN utils.TableDiscoveryUtils: 	 skipping database 'S*****' due to error reading tables: 由于数据库 'S*****' 离线，无法打开该数据库。
23/06/30 15:24:36 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:36 INFO reader.SourceReaderBase: Open Source Reader.
23/06/30 15:24:36 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:36 INFO eumerator.SqlServerChunkSplitter: Start splitting table BI.dbo.area into chunks...
23/06/30 15:24:37 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:37 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:37 WARN sqlserver.SqlServerDefaultValueConverter: Cannot parse column default value '(getdate())' to type 'datetime'. Expression evaluation is not supported.
23/06/30 15:24:37 INFO eumerator.SqlServerChunkSplitter: Use unevenly-sized chunks for table BI.dbo.area, the chunk size is 8096
23/06/30 15:24:37 INFO eumerator.SqlServerChunkSplitter: Split table BI.dbo.area into 1 chunks, time cost: 971ms.
23/06/30 15:24:37 INFO jdbc.JdbcConnection: Connection gracefully closed
23/06/30 15:24:37 INFO fetcher.SplitFetcher: Starting split fetcher 0
23/06/30 15:24:37 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:37 WARN sqlserver.SqlServerConnection: The 'server.timezone' option should be specified to avoid incorrect timestamp values in case of different timezones between the database server and this connector's JVM.
23/06/30 15:24:37 INFO history.DatabaseHistoryMetrics: Finished database history recovery of 0 change(s) in 1 ms
23/06/30 15:24:38 INFO fetcher.SplitFetcher: Finished reading from splits [BI.dbo.area:0]

Flink or Spark Version

spark：2.4.0

Java or Scala Version

java：8

Screenshots

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

cs3163077 avatar Jun 30 '23 07:06 cs3163077