sbt-databricks icon indicating copy to clipboard operation
sbt-databricks copied to clipboard

Published 20 hours ago • verified publisher icon databricks

Upgrading to use Databricks API v2.0

Open the-jackalope opened this issue 7 years ago • 5 comments

Hey @brkyvz !

I know the Databricks 2.0 API doesn't support all of the functionality of the 1.2 API, but would a PR that implemented some 2.0 functionality be welcome? Specifically the better cluster management tools that 2.0 supports would be very useful for our team.

I can't promise we'll put a PR in extremely soon, but I wanted to gauge whether that's something you'd be interested in merging into the project?

Thanks

the-jackalope avatar Jun 30 '17 16:06 the-jackalope

According to this, the library api is deprecated as of 2018, which correlates with my experience of being unable to run dbcUpload commands as part of our release process this year.

Is there any plans to upgrade this plugin to use the 2.0 api or is there a preferred workaround to get build artifacts into the Databricks ecosystem?

justinmills avatar Jan 23 '18 13:01 justinmills

I agree. I would love to use the sbt tool for deployment, but because of the deprecated REST API it's not possible. Can someone point out, what needs to be done? If it's not too much, I can invest some time to work on a pull request.

jpsimen avatar Jan 23 '18 13:01 jpsimen

I'm not sure this is possible in the 2.0 api. I've been looking over the 2.0 docs and cannot find where you can upload a jar anymore.

You might be able to use the DBFS api to upload the jar to DBFS and then attach it from there, but I can only find how to attach a jar from DBFS to a job, not an interactive cluster.

justinmills avatar Jan 23 '18 20:01 justinmills

Yeah, the biggest reason that I hadn't implemented a v2 version was because of a behavior change in the way libraries work through the REST API.

I would really appreciate it if you could submit a PR for it! It would have to follow the following pattern:

  1. Require you to upload the jar to DBFS
  2. Use that DBFS path to attach the jar to the cluster.

It will no longer show up in the Workspace.

brkyvz avatar Jan 23 '18 21:01 brkyvz

I submitted a PR (#45) to get dbcUpload working again under the 1.2 API. I verified that at least libraries/list and libraries/delete still work but not the other calls

assiotis avatar Jan 27 '18 17:01 assiotis