Spark in container?

[smomen]

Hi there - This seems the beginning of something very useful. For example, we'd like to make sure our libraries are tested in CI/CD in a databricks-like container before deploying in production. Two questions:

1. I get a 404 when trying to see published images @ https://cloud.docker.com/u/databricksruntime/repository/list, is that expected?
2. i ran the "standard" image as a container docker run -i -t databricksruntime/standard /bin/bash but don't seem to see spark or scala installed in the container (as I would expect in the runtime), is that wrong?

Thank you!

[evanye]

@smomen Sorry for missing this!

1. We updated the URL when we noticed. Check out https://docs.databricks.com/user-guide/clusters/custom-containers.html again.
2. These are base images, we actually inject spark and the databricks runtime when you launch the cluster.

Hope that answers your questions!

[sebbegg]

Sorry if I’m hijacking the issue - came here via google:

Is there a way to also install jars in a custom container so they end up in the spark classpath after launch?

[evanye]

@sebbegg I'll answer your question in https://github.com/databricks/containers/issues/22

[dipesh747]

@smomen Sorry for missing this!

1. We updated the URL when we noticed. Check out https://docs.databricks.com/user-guide/clusters/custom-containers.html again.
2. These are base images, we actually inject spark and the databricks runtime when you launch the cluster.

Hope that answers your questions!

How would a user inject the databricks runtime? I want to set up cicd on an ado agent and be able to test all the components I use in dbricks (e.g. mounting, dbutils, delta query etc.)

[evanye]

@dipesh747 the runtime is proprietary and we do not distribute it for local testing. Please discuss with your databricks account manager / support team if there are other options here.