百岁

[Question][PostgreSQL Source] which situation is fit for jdbcConf.isIncrement()=true && jdbcConf.isPolling()=false

1

### Search before asking - [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question. - [X] I had googled my question but i didn't get any help....

hudi 增量执行出错

2

hudi执行增量执行错误 ``` sql CREATE TABLE `stu` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '', `name` varchar(20) NOT NULL COMMENT '', `school` varchar(20) NOT NULL COMMENT '', `nickname` varchar(20) NOT...

mysql-> clickhouse ,启动过程中报错 ``` shell Caused by: com.qlangtech.tis.lang.TisException: java.lang.RuntimeException: java.lang.InstantiationException: java.lang.NoClassDefFoundError: com/qlangtech/tis/realtime/DefaultFlinkSourceHandle at com.qlangtech.tis.coredefine.module.action.CoreAction.doDeployIncrSyncChannal(CoreAction.java:575) ... 124 common frames omitted Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InstantiationException: java.lang.NoClassDefFoundError: com/qlangtech/tis/realtime/DefaultFlinkSourceHandle at com.qlangtech.plugins.incr.flink.TISFlinkClassLoaderFactory.lambda$buildClientLoaderFactory$1(TISFlinkClassLoaderFactory.java:120) at org.apache.flink.client.ClientUtils.buildUserCodeClassLoader(ClientUtils.java:87)...

hudi DeltaStreamer 执行过程中tpi包并发修改出错 ``` shell Caused by: java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441) at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1182) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)...

1. hudi中主键配置是支持复合主键的，目前TIS中只支持表拥有唯一主键（这样的要求是不合理的），需要在TIS控制台中允许用户选择复合主键的场景 2. 需要支持Hudi**complexkeygenerator** 这种分区策略以支持“ 大洲/国家/城市”，此分区是对应 数据表上 是三个字段https://hudi.apache.org/docs/key_generation#complexkeygenerator

 用户先在tidb 下设置数据源，然后更改dataXWriter类型之后该用mysql wirter类型，仍然可以使用tidb的datasource类型，点击**下一步** 之后就报以下错误： ``` shell 服务端发生异常，请联系系统管理员 ClassCastException: com.qlangtech.tis.plugin.ds.tidb.TiKVDataSourceFactory cannot be cast to com.qlangtech.tis.plugin.ds.BasicDataSourceFactory

目前实现基于： https://github.com/ivi-ru/flink-clickhouse-sink ``` xml ru.ivi.opensource flink-clickhouse-sink 1.3.3 ``` 不支持`delete`功能实现， 需要切换到，基于Clickhouse JDBC接口的实现上： ``` xml ru.yandex.clickhouse clickhouse-jdbc 0.3.1 ``` 可以参考streamx的实现 https://github.com/streamxhub/streamx-connector/tree/main/flink-connector-clickhouse 或者chunju的实现 https://github.com/DTStack/chunjun/tree/master/chunjun-connectors/chunjun-connector-clickhouse/src/main/java/com/dtstack/chunjun/connector/clickhouse

需要添加一种高级的DDL 语句同步方式，用户可以： 1. 选择 是否需要列同步 2. 可以控制DDL语句的模版

现在hudi增量执行过程中不支持物理删除功能，需要支持删除， 实现可参照： https://hudi.apache.org/docs/write_operations#delete

Dag调度任务现在是由jenkins actor模块处理的，需要在执行过程中对dag的并发度可以在界面上设置