RDB Shredder: validate that JSONs are conforming to Redshift limits

[BenFradet]

Given that only RDB Loader has knowledge of the targeted database, it makes sense that it enforces the database limits (e.g. 4mb for JSONs in Redshift).

[chuwy]

Hm, not sure it is possible in RDB Loader. Loader has an access only to folder structure.

[BenFradet]

Can't we try recovering from this issue? because it fails the whole load no matter what.

[chuwy]

As far as I know Redshift just silently truncates columns that exceeding a limit: https://github.com/snowplow/snowplow-rdb-loader/blob/master/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/loaders/RedshiftLoadStatements.scala#L180

[BenFradet]

those errors are not taken into account in max errors

[chuwy]

Sorry, I forgot the background on that issue. Is it a whole NDJSON line (entity in shredded type) that breaks a load? Or just a single column/property in a table/schema?

I think that RDB Loader shouldn't be in charge of swallowing/recovering from Redshift exceptions, but instead we should send this event into a bad row during shredding.

[BenFradet]

yes but different storage backends will have different properties that's what I meant in the original post.

so in Redshift, a JSON field can't be more than 4mb, what about postgres, bigquery, etc. and the shredder doesn't have any coupling regarding database.

[alexanderdean]

what about postgres, bigquery, etc. and the shredder doesn't have any coupling regarding database

The lack of coupling in Spark Shred sounds good in theory but I doubt it's practical (this thread is a good reason why).

[BenFradet]

@alexanderdean I think we should prioritize this, it's a huge drag for support atm when they have to recover.

[alexanderdean]

Agree, added to the next+1 milestone

[chuwy]

Turns out, we already constrain output https://github.com/snowplow/snowplow-rdb-loader/issues/103