Execute workflows with batch priority.

[nicolasrodriguez-mb]

Feature request:

It will be nice to execute workflows indicating batch priority. It should work as described here:

https://discuss.google.dev/t/is-it-possible-to-set-priority-of-dataform-execution-to-batch/155123/2?u=nicolas-mb

[kolina]

Do you want to have this ability in the GCP Dataform or in the Dataform CLI?

[nicolasrodriguez-mb]

it will be fabulous to achieve this.

Edit dataform.json Configuration:

Open your dataform.json file and add or modify the defaultConfig section:

{ "defaultConfig": { "bigquery": { "priority": "BATCH" } } } This sets BATCH as the default priority for all BigQuery operations in your Dataform project.

2. Configure in config Block (Granular Control):

For specific SQLX scripts, you can set the priority directly in the config block:

config { type: "table", bigquery: { priority: "BATCH" } }

-- Your SQL code here This allows you to control which scripts use BATCH priority.

[kolina]

Sure, but let me repeat my question: how do you run Dataform DAGs in production? Do you use Managed Dataform in GCP or you manage Dataform yourself and use the Dataform CLI?

[nicolasrodriguez-mb]

I use GCP

[kolina]

We're working on the support of configuring batch priority in the GCP Dataform at the moment. Initially it'll be supported as configuration when starting a new workflow invocation (as follow up we may add default configuration in dataform.json / workflow_settings.yaml)

[nicolasrodriguez-mb]

As a follow up, in GCP in the workflows, now has an option to unmark and leave the execution with batch priority. Pretty cool

[nicolasrodriguez-mb]

I believe the documentation in Datafom of GCP should highlight this.