clickhouse-scala-client icon indicating copy to clipboard operation
clickhouse-scala-client copied to clipboard

Clickhouse sink stream request to clickhouse instead of buffering in memory

Open ecyshor opened this issue 7 years ago • 0 comments

Currently the ClickhouseSink uses groupWithin to buffer the items which need to be inserted into clickhouse. This causes a larger memory footprint and also bursts of data being sent to clickhouse.

We should basically rotate the stream to a different http sink whenever the batch size or batch timeout has been reached.

ecyshor avatar Dec 18 '18 10:12 ecyshor