sql-runner
sql-runner copied to clipboard
snowplow
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
The project could use a real logger and logging could be refactored to make a bit more sense.
We import this package in https://github.com/snowplow/sql-runner/pull/192/files, do more neatly handle a case where we need to wrap errors. Once we have the space to do so, we should use that...
Sql-runner hit `- ERROR #XX000 basic_string::_M_construct null not valid` when attempting to run `REFRESH MATERIALIZED VIEW`. The view in question was constructed with LEFT joins, which it seems may not...
When SQL-runner can't find a file, it throws an error which contains the path it looked for the file, but not the line number of the relevant playbook. If possible,...
The Redshift/PG targets time out queries after 8 hours. If possible, add to Snowflake and BQ. Also would be nice to make it a configuration option in targets config.
It would be a nice convenience feature to be able to just set env vars for the playbook vars I want to use, then use some kind of flag in...
https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md This might be difficult because SQL Runner doesn't have the concept of input and output datasets as a first class citizen.
SQL-runner logs each step to stdout with timestamps, which is awesome. When doing gnarly debugging, or just longer-term logging of performance, it's potentially not ideal however, because if the query...
One of the challenges to writing a set of general SQL models is that some of the data we'd like to have models for isn't always present. So we're aiming...
This is currently something we do with our job orchestration tech - if there are multiple playbooks to be run in one job, a DAG dictates that job's execution. It...