flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-34883] Fix postgres uuid column as PK

Open eskabetxe opened this issue 9 months ago • 4 comments

A majority of Postgres databases use UUIDs as primary keys.

When we enable 'scan.incremental.snapshot.enabled = true', Flink-CDC will try to split into chunks. The splitTableIntoChunks function relies on the queryMinMax function, which fails when trying to calculate the MIN(UUID) and MAX(UUID), as that is not supported in Postgres.

This change those queries to cast the uuid to a String value.

What was done:

  • added uuid type for postgres mapped as string
  • for max and min queries, the uuid is casted to text: MIN(uuid::text)
  • for queries with >= and < we cast the param passed as: uuid >= (?)::uuid -- this way we can still use the index and only cast one value by query
  • changes on other databases are because we need to change an Splitter interface, to pass the Column instead of columnName

eskabetxe avatar Apr 29 '24 09:04 eskabetxe

@GOODBOY008 can you help here to find someone that can help review this?

eskabetxe avatar May 06 '24 14:05 eskabetxe

@leonardBang can you help here?

eskabetxe avatar Jun 03 '24 08:06 eskabetxe

Thanks @eskabetxe for the contribution, I'd like to find someone who is more familiar to Pg to review this PR.

leonardBang avatar Jun 03 '24 08:06 leonardBang

@loserwang1024 would you like to review this PR?

leonardBang avatar Jun 03 '24 08:06 leonardBang

@GOODBOY008 any chance to get this merged?

eskabetxe avatar Jul 14 '24 09:07 eskabetxe