dataform
dataform copied to clipboard
Dry run fails when table/view/assertion to be created does not exists
There are two possible errors relating to dry run:
Tables / views
- If the table created by a .sqlx file is not already materialised on BigQuery the dry run will fail due to dry run trying to
setMetadata
on an view/table that does not exist yet. Relevant lines of code
https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/commands/run.ts#L371
https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/dbadapters/bigquery.ts#L235-L239
In these lines metadata for the table/view that already does not exist will be undefined and trying to access .schema
attribute of undefined results in the error shown below
Assertion
Assertion dry run will face if the table it is trying to create does not already exists. Possibly due to trying to run the following query in the current code base which tries to select from a object that does not exist yet
https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/dbadapters/execution_sql.ts#L177
Minimal example to reproduce
- Add the following as a new .sqlx file to a dataform project
config {
type: "table",
schema: "dataset_name",
description: "any non empty description",
tags: ["TEST_DRY_RUN"]
}
SELECT 1 AS A
UNION ALL
SELECT 1 AS B
To recreate this error for view/assertion type, the type can be replaced by either of the two
- Run
dataform run --dry-run --tags="TEST_DRY_RUN"
dataform version: 3.0.2
and also tested by running it from source