clickhouse-kafka-connect
clickhouse-kafka-connect copied to clipboard
Expand Types of Sink Output to Match Clickhouse e.g. UInt64
Is your feature request related to a problem? Please describe. Clickhouse supports a wide range of data types, as does Protobuf (for serdes in Kafka Schema Registry). Whilst the data types that Clickhouse-Kafka-Connect Sink supports is a much smaller subset.
Recently, using Kafka in conjunction with Schema Registry, Clickhouse-Kafka-Connect Sink, and Protobuf I attempted to write Protobuf uint64 data to a Clickhouse UInt64 table column. I discovered (from the logs) that Clickhouse-Kafka-Connect Sink converts the Protobuf uint64 to int64, attempts to write that (and fails).
I don't believe this is an intuitive expectation, and indeed a significant limitation on the usefulness of the Clickhouse-Kafka-Connect Sink.
Describe the solution you'd like Please consider a much wider range of data type support to more closely match the range which Clickhouse & Protobuf supports. Most particularly, Clickhouse UInts of various widths, but indeed all types that can be matched or sensibly converted between, as available in Protobuf, and Avro.
(In this specific case I expected Protobuf uint64 to automatically convert to Clickhouse UInt64 type.)
Describe alternatives you've considered CustomSMT and Custom Converters. These come with considerable complexity, however.
Additional context Clickhouse-Kafka-Connect Sink - search the page for Supported Data Types: https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink
Clickhouse supported data types: https://clickhouse.com/docs/en/sql-reference/data-types
Protobuf supported (scalar) data types: https://protobuf.dev/programming-guides/proto3/#scalar
Avro supported data types: https://avro.apache.org/docs/1.11.1/specification/.