snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard

Redshift Loader // Narrowing from required to nullable would break loading

Open voropaevp opened this issue 3 years ago • 0 comments

If schema 1-0-0 had { ..."fieldX": {"type" : [ "string"] ...} and schema 1-0-1 had { ..."fieldX": "type" : [ "string", null]...}, then loader would not generate the ALTER table statement (and it would also be illegal for RedShift). Instead table comment would be updated to the new 1-0-1 version, while fieldX would remain required (not nullable).

Consequently, all batches where events have fieldX is NULL would fail to load.

voropaevp avatar Nov 29 '22 15:11 voropaevp