iceberg icon indicating copy to clipboard operation
iceberg copied to clipboard

Published 20 hours ago • verified publisher icon apache

Kafka Connect: Record converters and delta writers

Open bryanck opened this issue 1 year ago • 3 comments

This PR is the next stage in submitting the Iceberg Kafka Connect sink connector, and is a follow up to https://github.com/apache/iceberg/pull/8701 and https://github.com/apache/iceberg/pull/9466. It includes the record converters and the delta writers, with related config and tests. The delta writers are modeled after those in the Flink sink, and use equality deletes to apply deltas.

Still not included in the sink are the commit controller, integration tests, or docs, which will be added in follow up PRs. For reference, the current sink implementation can be found at https://github.com/tabular-io/iceberg-kafka-connect.

bryanck avatar Feb 04 '24 16:02 bryanck

Maybe @fqaiser94 is also interested in reviewing this :)

Fokko avatar Feb 21 '24 10:02 Fokko

A few comments/questions, but overall looks good. You might want to double check all the access levels for classes as I think we're generally overexposing.

danielcweeks avatar Feb 25 '24 18:02 danielcweeks

looks good to me overall, mostly questions/nits 👍

fqaiser94 avatar Feb 27 '24 14:02 fqaiser94

FYI, I removed the delta writers along with upsert and CDC related code.

bryanck avatar Feb 29 '24 00:02 bryanck

I believe I've addressed all comments, LMK if there is anything else.

bryanck avatar Mar 10 '24 01:03 bryanck

What's expected to be included in the next kafka-connect PR?

Asking b/c I'm doing some work on implementing zombie-fencing in tabular/iceberg-kafka-connect currently and we may want to wait to see what that looks like in terms of code changes before tackling the worker/coordinator portions of the code (in case it ends up being a large refactor).

fqaiser94 avatar Mar 12 '24 12:03 fqaiser94

What's expected to be included in the next kafka-connect PR?

Asking b/c I'm doing some work on implementing zombie-fencing in tabular/iceberg-kafka-connect currently and we may want to wait to see what that looks like in terms of code changes before tackling the worker/coordinator portions of the code (in case it ends up being a large refactor).

The next submission will be the coordinator piece. I have already addressed zombie fencing in that upcoming code, so I feel we can proceed with it for consideration.

bryanck avatar Mar 12 '24 14:03 bryanck

I was planning on merging this, unless someone wants to give more feedback, cc @fqaiser94 @danielcweeks

bryanck avatar Mar 21 '24 16:03 bryanck