MLFlow Error 409 when deploying --assets-only

[Stefanos-Arampatzis]

trafficstars

Expected Behavior

Deploy Jobs with --assets-only option

Current Behavior

MLFlow API Request 409 Conflict

Steps to Reproduce (for bugs)

[dbx][2022-11-03 12:30:40.370] 🔎 Deployment file is not provided, searching in the conf directory [dbx][2022-11-03 12:30:40.375] 💡 Auto-discovery found deployment file conf/deployment.json [dbx][2022-11-03 12:30:40.376] 🆗 Deployment file conf/deployment.json exists and will be used for deployment [dbx][2022-11-03 12:30:40.377] Starting new deployment for environment dev [dbx][2022-11-03 12:30:40.378] Using profile provided from the project file [dbx][2022-11-03 12:30:40.378] Found auth config from provider EnvironmentVariableConfigProvider, verifying it [dbx][2022-11-03 12:30:40.379] Found auth config from provider EnvironmentVariableConfigProvider, verification successful [dbx][2022-11-03 12:30:44.897] Since v0.7.0 environment configurations should be nested under environments section.

            Please nest environment configurations under this section to avoid potential issues while using "build"
            configuration directive.

[dbx][2022-11-03 12:30:44.899] No build logic defined in the deployment file. Default pip-based build logic will be used. [dbx][2022-11-03 12:30:44.903] Usage of jobs keyword in deployment file is deprecated. Please use workflows instead (simply rename this section to workflows). [dbx][2022-11-03 12:30:44.906] Workflows ['add-on-chanel-AT', 'add-on-chanel-PL', 'add-on-PL'] were selected for further operations [dbx][2022-11-03 12:30:44.907] Following the provided build logic [dbx][2022-11-03 12:30:44.908] 🐍 Building a Python-based project [dbx][2022-11-03 12:30:46.262] ✅ Python-based project build finished [dbx][2022-11-03 12:30:46.264] Locating package file [dbx][2022-11-03 12:30:46.265] Package file located in: dist/ds_recommenders-1.2.9-py3-none-any.whl [dbx][2022-11-03 12:30:47.221] Starting the traversal process [dbx][2022-11-03 12:30:47.222] Processing libraries for workflow add-on-chanel-AT [dbx][2022-11-03 12:30:47.223] ✅ Processing libraries for workflow add-on-chanel-AT - done [dbx][2022-11-03 12:30:47.224] Processing libraries for workflow add-on-chanel-PL [dbx][2022-11-03 12:30:47.225] ✅ Processing libraries for workflow add-on-chanel-PL - done [dbx][2022-11-03 12:30:47.225] Processing libraries for workflow add-on-PL [dbx][2022-11-03 12:30:47.226] ✅ Processing libraries for workflow add-on-PL - done [dbx][2022-11-03 12:30:47.227] ⬆ Uploading local file src/jobs/add_on_products/add_on_chanel/chanel_AT.py [dbx][2022-11-03 12:30:50.412] ✅ Uploading local file src/jobs/add_on_products/add_on_chanel/chanel_AT.py [dbx][2022-11-03 12:30:50.414] ⬆ Uploading local file src/jobs/add_on_products/add_on_chanel/chanel_AT.py ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/dbx/comma │ │ nds/deploy.py:157 in deploy │ │ │ │ 154 │ │ │ │ wf_manager = WorkflowDeploymentManager(api_client, ele │ │ 155 │ │ │ │ wf_manager.apply() │ │ 156 │ │ else: │ │ ❱ 157 │ │ │ adjuster.traverse(deployable_workflows) │ │ 158 │ │ │ if not _assets_only: │ │ 159 │ │ │ │ wf_manager = WorkflowDeploymentManager(api_client, dep │ │ 160 │ │ │ │ wf_manager.apply() │ │ │ │ /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/dbx/api/a │ │ djuster/adjuster.py:185 in traverse │ │ │ │ 182 │ def traverse(self, workflows: Union[WorkflowList, List[str]]): │ │ 183 │ │ dbx_echo("Starting the traversal process") │ │ 184 │ │ self.property_adjuster.library_traverse(workflows, self.additi │ │ ❱ 185 │ │ self.property_adjuster.file_traverse(workflows, self.file_adju │ │ 186 │ │ self.property_adjuster.property_traverse(workflows) │ │ 187 │ │ self.property_adjuster.cluster_policy_traverse(workflows) │ │ 188 │ │ dbx_echo("Traversal process finished, all provided references │ │ │ │ /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/dbx/api/a │ │ djuster/adjuster.py:168 in file_traverse │ │ │ │ 165 │ │ for element, parent, index in self.traverse(workflows): │ │ 166 │ │ │ if isinstance(element, str): │ │ 167 │ │ │ │ if element.startswith("file://") or element.startswith │ │ ❱ 168 │ │ │ │ │ file_adjuster.adjust_file_ref(element, parent, ind │ │ 169 │ │ 170 │ │ 171 class Adjuster: │ │ │ │ /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/dbx/api/a │ │ djuster/mixins/file_reference.py:12 in adjust_file_ref │ │ │ │ 9 │ │ self._uploader = file_uploader │ │ 10 │ │ │ 11 │ def adjust_file_ref(self, element: str, parent: Any, index: Any): │ │ ❱ 12 │ │ _uploaded = self._uploader.upload_and_provide_path(element) │ │ 13 │ │ self.set_element_at_parent(_uploaded, parent, index) │ │ 14 │ │ │ │ /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/dbx/utils │ │ /file_uploader.py:59 in upload_and_provide_path │ │ │ │ 56 │ │ │ self._verify_fuse_support() │ │ 57 │ │ │ │ 58 │ │ dbx_echo(f":arrow_up: Uploading local file {local_file_path}") │ │ ❱ 59 │ │ self.upload_file(local_file_path) │ │ 60 │ │ dbx_echo(f":white_check_mark: Uploading local file {local_file │ │ 61 │ │ return self._postprocess_path(local_file_path, as_fuse) │ │ 62 │ │ /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/mlflow/ut │ │ ils/rest_utils.py:199 in http_request_safe │ │ │ │ 196 │ Wrapper around http_request that also verifies that the reques │ │ 197 │ """ │ │ 198 │ response = http_request(host_creds=host_creds, endpoint=endpoint, │ │ ❱ 199 │ return verify_rest_response(response, endpoint) │ │ 200 │ │ 201 │ │ 202 def verify_rest_response(response, endpoint): │ │ │ │ /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/mlflow/ut │ │ ils/rest_utils.py:212 in verify_rest_response │ │ │ │ 209 │ │ │ │ endpoint, │ │ 210 │ │ │ │ response.status_code, │ │ 211 │ │ │ ) │ │ ❱ 212 │ │ │ raise MlflowException( │ │ 213 │ │ │ │ "%s. Response body: '%s'" % (base_msg, response.text), │ │ 214 │ │ │ │ error_code=get_error_code(response.status_code), │ │ 215 │ │ │ ) │ ╰──────────────────────────────────────────────────────────────────────────────╯ MlflowException: API request to endpoint /dbfs/Shared/ds_recommenders/projects/ds_recommenders_experiments/8cfd06c9088742 8b97e6371f9225de5a/artifacts/src/jobs/add_on_products/add_on_chanel/chanel_AT.py failed with error code 409 != 200. Response body: '

HTTP ERROR: 409

Problem accessing /dbfs/Shared/ds_recommenders/projects/ds_recommenders_experiments/8cfd06c9088742 8b97e6371f92[25](https://github.com/Flaconi/DS_Recommenders/actions/runs/3385675925/jobs/5624102075#step:11:26)de5a/artifacts/src/jobs/add_on_products/add_on_chanel/chanel_AT.py . Reason:

    File already exists, cannot overwrite: 
'/Shared/ds_recommenders/projects/ds_recommenders_experiments/8cfd06c908874
[28](https://github.com/Flaconi/DS_Recommenders/actions/runs/3385675925/jobs/5624102075#step:11:29)b97e6[37](https://github.com/Flaconi/DS_Recommenders/actions/runs/3385675925/jobs/5624102075#step:11:38)1f[92](https://github.com/Flaconi/DS_Recommenders/actions/runs/3385675925/jobs/5624102075#step:11:93)25de5a/artifacts/src/jobs/add_on_products/add_on_chanel/chanel_AT.p
y'

---

' Error: Process completed with exit code 1.

Context

Updated few jobs today using the latest dbx version, and at the jobless deployment cicd step I get the error above. MLFlow is only used to define a specific experiment path. No path related updates or changes here!

Your Environment

• dbx version used: 0.8.x
• Databricks Runtime version: 10.4 LTS (standard or ML)
• Python version: 3.8.11

[renardeinside]

hi @Stefanos-Arampatzis , could you please provide the section of the deployment file where this file is referencedsrc/jobs/add_on_products/add_on_chanel/chanel_AT.py?

[Stefanos-Arampatzis]

hi @renardeinside here it is

"spark_python_task": {
    "python_file": "file://src/jobs/add_on_products/add_on_chanel/chanel_AT.py",
    ...
}

[renardeinside]

hi @Stefanos-Arampatzis , unfortunately I cannot reproduce this issue in any of the local setups. Is it still an issue you're running into?

[Stefanos-Arampatzis]

@renardeinside I just tried again to deploy and still get the same error. I now have the same behaviour in 3 different repos. I could enforce 0.7.6 to be installed, until we can find what's the issue. Thanks for the support

[Stefanos-Arampatzis]

@renardeinside I tried the latest version, 0.8.4 and the error persists. It works with 0.7.6 so doesn't seem like a 4xx error on our side. Please let me know if you would need more context to debug.