dagster
dagster copied to clipboard
dagster-io
feat(databricks): Show Databricks job URL in Dagit UI
Summary & Motivation
When a Databricks job is launched, and we need to inspect the Databricks cluster (such as event log or memory usage), we need the Databricks URL, which is currently cumbersome to find (basically go to the Databricks jobs list page and cmd-f to find the relevant job URL).
A simple solution would be to display the Databricks job URL in the Dagit UI logs itself.
How I Tested These Changes
Need some help on how to test this.
@swaroopch is attempting to deploy a commit to the Elementl Team on Vercel.
A member of the Team first needs to authorize it.
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
The latest updates on your projects. Learn more about Vercel for Git ↗︎
2 Ignored Deployments
| Name | Status | Preview | Updated |
|---|---|---|---|
| dagit-storybook | ⬜️ Ignored (Inspect) | Jul 14, 2022 at 8:07PM (UTC) | |
| dagster-oss-cloud-consolidated | ⬜️ Ignored (Inspect) | Jul 14, 2022 at 8:07PM (UTC) |
Hi @swaroopch ! Thanks for the contribution. This implementation seems totally reasonable to me. It looks like you'll need to update some tests to mock out that API call:
| [2022-07-28T21:42:35Z] FAILED dagster_databricks_tests/test_databricks.py::test_databricks_wait_for_run
| [2022-07-28T21:42:35Z] FAILED dagster_databricks_tests/test_solids.py::test_run_create_databricks_job_solid[create_databricks_job_solid]
| [2022-07-28T21:42:35Z] FAILED dagster_databricks_tests/test_solids.py::test_run_create_databricks_job_solid[create_databricks_job_op]
I think that would look something like adding a @mock.patch("dagster_databricks.databricks.DatabricksClient.get_run", mock.MagicMock(return_value={"run_page_url": "www.foo.com"}), but I didn't test that myself.
You'll also want to run make black in the root directory once you're done, to make sure everything is formatted correctly. See: https://docs.dagster.io/community/contributing#developing-dagster for some additional instructions :)
