elementary icon indicating copy to clipboard operation
elementary copied to clipboard

Value too long in compiled_sql

Open Maayan-s opened this issue 3 years ago • 1 comments

We don't want to create unnecessarily large columns, as this is not a good practice in Redshift: https://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-smallest-column-size.html

However, queries might get very long, so for that case, the 16K limit we have now is not enough.

The suggested solution would be:

  1. Creating a new data type, max_long_string
  2. Changing the type for 'compiled_sql' to max_long_string in the relevant empty queries
  3. Validate that the input string is not larger than this size, and trim it if it is
  4. Test migration on the incremental tables (upgrade and see it works)

Maayan-s avatar Sep 04 '22 19:09 Maayan-s

https://github.com/elementary-data/dbt-data-reliability/pull/97

Maayan-s avatar Sep 20 '22 20:09 Maayan-s

Released on 0.4.12 of the dbt pakage!

Maayan-s avatar Sep 23 '22 17:09 Maayan-s