elementary
elementary copied to clipboard
Value too long in compiled_sql
We don't want to create unnecessarily large columns, as this is not a good practice in Redshift: https://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-smallest-column-size.html
However, queries might get very long, so for that case, the 16K limit we have now is not enough.
The suggested solution would be:
- Creating a new data type, max_long_string
- Changing the type for 'compiled_sql' to max_long_string in the relevant empty queries
- Validate that the input string is not larger than this size, and trim it if it is
- Test migration on the incremental tables (upgrade and see it works)
https://github.com/elementary-data/dbt-data-reliability/pull/97
Released on 0.4.12 of the dbt pakage!