"edit_mode" attribute is missing in Job class for Python for Databricks Asset Bundles

[henryhueske]

Describe the issue

In the REST API request to create a new job (see here), it is possible to set the parameter edit_mode for a job. However, the related attribute is not available in the Job class in Python for Databricks Asset Bundles, see here.

From my point of view, this attribute is very important to be included, because deploying jobs with the default edit mode UI_LOCKED is practically unusable in the day-to-day business of job runs, because it occurs often that schedules have to paused or other things have to be done in order to react to job failures or maintenance/update tasks. And typically, deployment via Databricks Asset bundles is used for all environments and especially in development environments, it does not make sense to have jobs locked in the UI.

Additionally, as I tried to add the edit_mode parameter manually in a job configuration, e.g. by supplying the related YAML definition using the from_dict method, it still didn't work, because I got the following error:

Traceback (most recent call last):
  File ".../python3.10/site-packages/databricks/bundles/build.py", line 94, in _load_resources_from_input
    resource = _transform(tpe.resource_type, resource_dict)
  File ".../python3.10/site-packages/databricks/bundles/core/_transform.py", line 176, in _transform
    raise ValueError(f"Unexpected field '{key}' for class {cls.__name__}")
ValueError: Unexpected field 'edit_mode' for class Job

The only workaround that helped solving the issue was to add a post step after the deployment using the Python Databricks SDK in order to change the edit_mode of all jobs.

Configuration

Setup a simple job configuration with Python for Databricks Asset Bundles and try to add the edit_mode parameter (either via YAML or using the class definitions)

Steps to reproduce the behavior

After configuration, try to run the deployment using YAML configuration.

Expected Behavior

Attribute edit_mode is available in the Job class

Actual Behavior

Attribute edit_mode is not available in the Job class

OS and CLI version

Ubuntu 24.04, Python 3.10, Python for Asset Bundles 0.266.0

Is this a regression?

No.

[pietern]

Thanks for creating the issue.

This is hidden by design. Our intent is to have a visual cue that any changes you make to a job that is managed by a DAB are clobbered on the next deployment. To break the lock, you can acknowledge this by clicking the disconnect button in the UI and then making any change you want (as long as you have permission to do so).

[henryhueske]

@pietern: Thanks for your quick response and for the explanation. I did some further tests and now it makes sense to me. I wasn't aware that the job is re-connected to the bundle with the next deployment after it was disconnected. I thought it is completely disconnected from the bundle.

Nevertheless, from my point of view, it should be up to the developer/user of the library to set the edit_mode to the value that he/she thinks makes most sense for him/her. I am okay with having UI_LOCKED as default value, but as the field is available via the SDK and the REST API, it should be also usable via the Python library for Databricks Asset Bundles. BTW: In the docs it is not mentioned that UI_LOCKED is the default mode.

[peter-ocelka]

@pietern: Would it be possible to update the documentation ; i.e. remove the edit_mode if it is not supported by bundles? The same "not working" behavior is also in yaml DAB definition.

I have to aslo agree with @henryhueske. There are situations when you just need to do some changes on the job. In our cases, we also have a scenario that from time to time we need to pause the schedule due to outages in upstream systems and running job causes that we get a lot incidents that something is wrong with the job.

From my perspective, both sides have some valid arguments. Maybe you could introduce new CLI parameter to override this behavior (ideally if it could support silent override for CI/CD purposes).

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.