dbt-athena icon indicating copy to clipboard operation
dbt-athena copied to clipboard

dbt seed fails when too long content

Open plektra opened this issue 2 years ago • 0 comments

When trying to add a seed file that's too long for Athena, the operation fails because it exceeds the query string length hard limit of 262144 bytes. The error message is somewhat like this:

An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: 1 validation error detected: Value 'insert into database.table (col1, col2, ...) values (........) at 'queryString' failed to satisfy constraint: Member must have length less than or equal to 262144

Perhaps it's not a good use of seeds when the table size exceeds single query lenght limits, but at least FYI for those wondering about the same issue. I worked around this issue by introducing my static table as a new Glue table from S3 data.

plektra avatar Oct 19 '22 06:10 plektra