clickhouse_sinker
clickhouse_sinker copied to clipboard
housepower
Easily load data from kafka to ClickHouse
I'll create a "Who uses this?" small list in the README. If you would like your company to be on this list, please mention it on this issue. For example,...
i find a scenario when finish a pressure test clickhouse_sinker container got killed by the system oom killer as the memory consumption overflow the memory limit set in deployment spec....
  
``` {"level":"warn","ts":"2022-01-20T11:31:08.029+0800","msg":"unable to commit offset for topic partition","group":"metric-010","topic":"perf-metric","partition":2,"commit_from":29746158,"commit_to":29747354,"commit_epoch":-1,"error_code":25} {"level":"debug","ts":"2022-01-20T11:31:08.029+0800","msg":"updated committed","group":"metric-010","to":"perf-metric[]"} {"level":"debug","ts":"2022-01-20T11:31:08.029+0800","msg":"left CommitOffsetsSync","group":"metric-010"} {"level":"fatal","ts":"2022-01-20T11:31:08.029+0800","msg":"Batch.Commit failed with permanent error","task":"sink-metric","error":"UNKNOWN_MEMBER_ID: The coordinator is not aware of this member.","stacktrace":"github.com/housepower/clickhouse_sinker/output.(*ClickHouse).loopWrite\n\t/data01/zhichyu/clickhouse_sinker/output/clickhouse.go:178\ngithub.com/housepower/clickhouse_sinker/output.(*ClickHouse).Send.func1\n\t/data01/zhichyu/clickhouse_sinker/output/clickhouse.go:99\ngithub.com/housepower/clickhouse_sinker/util.(*WorkerPool).wokerFunc\n\t/data01/zhichyu/clickhouse_sinker/util/workerpool.go:59"} ``` [sinker_metric.log](https://github.com/housepower/clickhouse_sinker/files/7904196/sinker_metric.log)
https://aseigneurin.github.io/2018/08/02/kafka-tutorial-4-avro-and-schema-registry.html https://github.com/confluentinc/schema-registry/blob/master/bin/kafka-avro-console-consumer https://github.com/twmb/franz-go/issues/110
在最新发布版本(2.5.2)中，配置分布表为目的表时存在错误的配置检查 在system.tables 中检查分布表时，匹配create_table_query字段匹配建表参数时，程序使用了配置中的分布表的名称而不是分布表关联的本地表名称匹配建表参数导致记录匹配失败。
