Defining Serverless version for notebook jobs

[ikhudur]

Describe the bug

I cannot define the version of the serverless cluster to use when running a dbt python model. I have tried several different ways, butit always defaults to environment version 1.

The issue seems to be in how the fields are defined, and I have identified possible locations where it should be updated (I am however not sure why the code was written as it is, so I might be missing something).

Issues related to this: https://github.com/databricks/dbt-databricks/issues/1009 https://github.com/databricks/dbt-databricks/issues/1055 https://github.com/databricks/dbt-databricks/issues/1195

Steps To Reproduce

Create a simple dbt python model (it can return some values). Try to setup the yaml config to define environment version, here is an example:

version: 2
models:
  - name: my_model
    config:
      materialized: table
      submission_method: serverless_cluster
      environments:
        - environment_key: default
          spec:
            environment_version: "3"
      python_job_config:
        name: my_model_single_run
        additional_task_settings: {"environment_key": "default"}

Expected behavior

The single notebook job run that is created and submitted uses serverless verion 3

Screenshots and log output

System information

The output of dbt --version:

dbt Cloud CLI - 0.40.7 (3aa8c1ef8d89fce3bf5750f1a37255a5321f0a5a 2025-10-06T18:56:38Z)

The operating system you're using: WSL2 Ubuntu-24.04

The output of python --version:

Additional context

I think the issue with the code starts with the class PythonJobConfig here : https://github.com/databricks/dbt-databricks/blob/38a89108c49a09f3e428e08dae634ae563ac7824/dbt/adapters/databricks/python_models/python_config.py#L1-L92

In is missing the definition for the field environments, which appears as in Databricks REST API

Furthermore, in the part of the code that puts the PythonJobDetails together it can be seen that it only sets the environments field IF environment_deps is set and it cannot find the field environments in python_job_config (which it never will, as it is not defined in the class PythonJobConfig):

https://github.com/databricks/dbt-databricks/blob/38a89108c49a09f3e428e08dae634ae563ac7824/dbt/adapters/databricks/python_models/python_submissions.py#L272-L326

And here are 3 issues: The first, is that it defaults to version 1 for me.

The second and third, are that serverless version 2 is hardcoded and the depracted client field is used (see the Databricks REST API documentation) https://github.com/databricks/dbt-databricks/blob/38a89108c49a09f3e428e08dae634ae563ac7824/dbt/adapters/databricks/python_models/python_submissions.py#L312

So, my hunch is that a quick fix is to do something like this:

class PythonJobConfig(BaseModel):
    """Pydantic model for config found in python_job_config."""

    name: Optional[str] = None
    grants: dict[str, list[dict[str, str]]] = Field(exclude=True, default_factory=dict)
    existing_job_id: str = Field("", exclude=True)
    environments: Optional[list[dict[str, Any]]] # Add this field
    post_hook_tasks: list[dict[str, Any]] = Field(exclude=True, default_factory=list)
    additional_task_settings: dict[str, Any] = Field(exclude=True, default_factory=dict)

    class Config:
        extra = "allow"

But I am not entirely sure

[benc-db]

Please take a look at #1286 and if possible, verify that it fixes the issue.