Container for specific runtime versions

[casparjespersen]

I developing Python applications to be run on Databricks, I am looking towards Databricks Containers in ensuring that I develop on an environment as close to the runtime environment as possible (Spark libraries and connectors, etc.). However, when running the containers I cannot find any of the runtime versions (7.3, 7.4, etc.), and the latest image is not packed with any of the libraries.

Where can I find Docker images pre-loaded with Spark and associated jar-packages/libraries for etc. DB Runtime 7.3 LTS?

[guiferviz]

This may be helpful for you: https://github.com/databricks/containers/issues/11#issuecomment-526416129

These are base images, we actually inject spark and the databricks runtime when you launch the cluster.

So, as far as I know, you can configure everything you want in your image except all the things related to spark/databricks runtime.

[evanye]

What are you trying to package? For jars, you can drop them into /databricks/jars inside the container, and the runtime should pick it up. For python libraries, see the included conda images for how to manage your own env.