debezium-server-iceberg icon indicating copy to clipboard operation
debezium-server-iceberg copied to clipboard

How does this consumer handle Toasted Values in PostgreSQL?

Open ndrluis opened this issue 1 year ago • 1 comments

Hello,

I'm implementing CDC with Iceberg, and currently, I have several columns that are JSONB and in some UPDATE events, when they are not updated, they end up returning with the placeholder indicating that it is a TOASTED column.

Is there any way to handle these columns on the consumer side?

https://debezium.io/blog/2019/10/08/handling-unchanged-postgres-toast-values/

ndrluis avatar May 29 '23 13:05 ndrluis

Hi @ndrluis

i cant think of a way to handle it efficiently. current implementation is deleting full row and overriding with new one fully using iceberg api. this means data value will be lost/overridden.

probably best to handle them with post processing. running it with append only mode.

ismailsimsek avatar May 31 '23 12:05 ismailsimsek