Download artefacts from Databricks Job

[meretri]

trafficstars

Describe the bug

Actually it is more of a question than a bug. I am using dbt core, not dbt cloud together with Databricks. I have been trying to download the artefacts (manifest.json, ...) when we run a dbt Job using Databricks Workflow from the cluster after the job has finished. You mention in the documentation that you can achieve this by downloading the artefacts using the jobs API. I realised that the provided commands are not working anymore with the latest Databricks CLI Version (v0.212.0). I was also not able to successfully download the artefacts using databricks jobs get-run-output ID, because the return is missing the "dbt_output" field as documented here.

Do you know how to do this?

[benc-db]

You should be able to download them from the Workflow UI after the job has completed, though I have seen instances where the button doesn't show up. If the commands no longer work with the Databricks CLI, I recommend filing a bug with that project.

[KoningJasper]

It doesn't work if you have encryption enabled on the bucket. This limitation should go away in the next couple weeks.

[towenseb]

@KoningJasper any update on this? I'm also getting the behavior described by meretri and I checked just now and am still unable to get the artifacts.

[KoningJasper]

@towenseb we rolled this out but it caused some issues so we're fixing those and then rolling out again. So ETA applies, couple of weeks.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.

[towenseb]

unstale

[philk1991]

Also impacted by this. Are there any workarounds?

[meretri]

@KoningJasper any news on this topic? or do you have any recommendations for workarounds?

[benc-db]

This issue is best addressed by filing tickets with Databricks. The adapter does not control whether you can download artifacts from a job.