clickhouse-kafka-connect icon indicating copy to clipboard operation
clickhouse-kafka-connect copied to clipboard

Expand Types of Sink Output to Match Clickhouse e.g. UInt64

Open PaulST2 opened this issue 5 months ago • 2 comments

Is your feature request related to a problem? Please describe. Clickhouse supports a wide range of data types, as does Protobuf (for serdes in Kafka Schema Registry). Whilst the data types that Clickhouse-Kafka-Connect Sink supports is a much smaller subset.

Recently, using Kafka in conjunction with Schema Registry, Clickhouse-Kafka-Connect Sink, and Protobuf I attempted to write Protobuf uint64 data to a Clickhouse UInt64 table column. I discovered (from the logs) that Clickhouse-Kafka-Connect Sink converts the Protobuf uint64 to int64, attempts to write that (and fails).

I don't believe this is an intuitive expectation, and indeed a significant limitation on the usefulness of the Clickhouse-Kafka-Connect Sink.

Describe the solution you'd like Please consider a much wider range of data type support to more closely match the range which Clickhouse & Protobuf supports. Most particularly, Clickhouse UInts of various widths, but indeed all types that can be matched or sensibly converted between, as available in Protobuf, and Avro.

(In this specific case I expected Protobuf uint64 to automatically convert to Clickhouse UInt64 type.)

Describe alternatives you've considered CustomSMT and Custom Converters. These come with considerable complexity, however.

Additional context Clickhouse-Kafka-Connect Sink - search the page for Supported Data Types: https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink

Clickhouse supported data types: https://clickhouse.com/docs/en/sql-reference/data-types

Protobuf supported (scalar) data types: https://protobuf.dev/programming-guides/proto3/#scalar

Avro supported data types: https://avro.apache.org/docs/1.11.1/specification/.

PaulST2 avatar Sep 19 '24 21:09 PaulST2