tis
tis copied to clipboard
datavane
Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
hudi执行增量执行错误 ``` sql CREATE TABLE `stu` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '', `name` varchar(20) NOT NULL COMMENT '', `school` varchar(20) NOT NULL COMMENT '', `nickname` varchar(20) NOT...
mysql-> clickhouse ,启动过程中报错 ``` shell Caused by: com.qlangtech.tis.lang.TisException: java.lang.RuntimeException: java.lang.InstantiationException: java.lang.NoClassDefFoundError: com/qlangtech/tis/realtime/DefaultFlinkSourceHandle at com.qlangtech.tis.coredefine.module.action.CoreAction.doDeployIncrSyncChannal(CoreAction.java:575) ... 124 common frames omitted Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InstantiationException: java.lang.NoClassDefFoundError: com/qlangtech/tis/realtime/DefaultFlinkSourceHandle at com.qlangtech.plugins.incr.flink.TISFlinkClassLoaderFactory.lambda$buildClientLoaderFactory$1(TISFlinkClassLoaderFactory.java:120) at org.apache.flink.client.ClientUtils.buildUserCodeClassLoader(ClientUtils.java:87)...
hudi DeltaStreamer 执行过程中tpi包并发修改出错 ``` shell Caused by: java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441) at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1182) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)...
order_info表设置分区策略为slashEncodedDay，字段选取create_time，同时设置pt为分区字段，触发执行成功。分别使用hive和presto查询表的分区情况和count数都无误，然后改使用spark-sql查询分区情况时会提示如下信息： ``` spark-sql> show partitions order_info; org.apache.spark.sql.AnalysisException: SHOW PARTITIONS is not allowed on order_info since its partition metadata is not stored in the Hive metastore. To import this information into...
在tis-web-config/config.properties文件中改用mysql数据库，如下 ``` project.name=TIS runtime=daily tis.datasource.type=mysql tis.datasource.url=127.0.0.1 tis.datasource.port=3306 tis.datasource.username=root tis.datasource.password=root tis.datasource.dbname=tis_console zk.host=127.0.0.1:2181/tis/cloud assemble.host=127.0.0.1 tis.host=127.0.0.1 ``` 启动前后端服务，访问http://localhost:4200时后端服务会抛错，如下： Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at...
1. hudi中主键配置是支持复合主键的，目前TIS中只支持表拥有唯一主键（这样的要求是不合理的），需要在TIS控制台中允许用户选择复合主键的场景 2. 需要支持Hudi**complexkeygenerator** 这种分区策略以支持“ 大洲/国家/城市”，此分区是对应 数据表上 是三个字段https://hudi.apache.org/docs/key_generation#complexkeygenerator
TIS 增量同步很慢，一个表需要10秒+多以上才能处理完，希望优化下这个速度
 用户先在tidb 下设置数据源，然后更改dataXWriter类型之后该用mysql wirter类型，仍然可以使用tidb的datasource类型，点击**下一步** 之后就报以下错误： ``` shell 服务端发生异常，请联系系统管理员 ClassCastException: com.qlangtech.tis.plugin.ds.tidb.TiKVDataSourceFactory cannot be cast to com.qlangtech.tis.plugin.ds.BasicDataSourceFactory
目前实现基于： https://github.com/ivi-ru/flink-clickhouse-sink ``` xml ru.ivi.opensource flink-clickhouse-sink 1.3.3 ``` 不支持`delete`功能实现， 需要切换到，基于Clickhouse JDBC接口的实现上： ``` xml ru.yandex.clickhouse clickhouse-jdbc 0.3.1 ``` 可以参考streamx的实现 https://github.com/streamxhub/streamx-connector/tree/main/flink-connector-clickhouse 或者chunju的实现 https://github.com/DTStack/chunjun/tree/master/chunjun-connectors/chunjun-connector-clickhouse/src/main/java/com/dtstack/chunjun/connector/clickhouse