peerdb icon indicating copy to clipboard operation
peerdb copied to clipboard

Convert PG Composite Types to Tuples in CH

Open dungeon-master-666 opened this issue 5 months ago • 4 comments

Currently custom composite types from postgres are saved as strings in CH which makes it inconvenient to access/filter/sort/etc them. This PR adds functionality converting composite types to Tuple.

  1. Nested composite types and arrays of composites are supported
  2. Clickhouse doesn't have Nullable(Tuple(...)) so instead every field of tuple is created Nullable (if PEERDB_NULLABLE is enabled)
  3. Apparently this feature should be wrapped in setting to make existing mirrors won't break.
  4. I tested only the direct scenario, PG -> CH mirror. PG -> PG and others were not tested

dungeon-master-666 avatar Jul 10 '25 16:07 dungeon-master-666

#3066 introduced versioning as a simpler alternative to settings for maintaining backwards compatibility

serprex avatar Jul 10 '25 23:07 serprex

wanted to get to bringing this to a point we can merge, but been falling behind this past month

serprex avatar Aug 19 '25 14:08 serprex

wanted to get to bringing this to a point we can merge, but been falling behind this past month

I'm doing a bit of refactoring right now and fixing several issues. Will be ready for review in a day or 2.

dungeon-master-666 avatar Aug 19 '25 15:08 dungeon-master-666

Hi @serprex @heavycrystal, it's ready for reviewing. I've added a new version CompositeTypeAsTuple for backward compatibility and fixed several issues with arrays of composites and nested tuples.

dungeon-master-666 avatar Aug 22 '25 14:08 dungeon-master-666