dataform icon indicating copy to clipboard operation
dataform copied to clipboard

Dry run fails when table/view/assertion to be created does not exists

Open ashish10alex opened this issue 6 months ago • 0 comments

There are two possible errors relating to dry run:

Tables / views

  1. If the table created by a .sqlx file is not already materialised on BigQuery the dry run will fail due to dry run trying to setMetadata on an view/table that does not exist yet. Relevant lines of code

https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/commands/run.ts#L371

https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/dbadapters/bigquery.ts#L235-L239

In these lines metadata for the table/view that already does not exist will be undefined and trying to access .schema attribute of undefined results in the error shown below

CleanShot 2024-08-12 at 19 16 41@2x


Assertion

Assertion dry run will face if the table it is trying to create does not already exists. Possibly due to trying to run the following query in the current code base which tries to select from a object that does not exist yet

https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/dbadapters/execution_sql.ts#L177


Minimal example to reproduce

  1. Add the following as a new .sqlx file to a dataform project
config {
  type: "table",
  schema: "dataset_name",
  description: "any non empty description",
  tags: ["TEST_DRY_RUN"]
}


SELECT 1 AS A
UNION ALL
SELECT 1 AS B

To recreate this error for view/assertion type, the type can be replaced by either of the two

  1. Run
    dataform run --dry-run --tags="TEST_DRY_RUN"
    

dataform version: 3.0.2 and also tested by running it from source

ashish10alex avatar Aug 12 '24 18:08 ashish10alex