Carsten Behring

Regarding the problem of "interactive" asking for something: I have seen this from time to time, and there are usually specific command line options to prevent them. But it is...

The closetsI found was python + java: https://github.com/jupyter/docker-stacks/blob/main/pyspark-notebook/Dockerfile But is has a lot of other staff as well, so gets big.

an other pproach is to use "devcontainers". This separates "base image" and "features added to it". I made a working one here: https://github.com/behrica/clojure-datascience-devcontainer

This issue makes it impossible to use geni with databricks.

I digged into it. While I was always wondering, if setting options here : https://github.com/zero-one-group/geni/blob/e6eaeb937d7598648e1986928d35f5bea336441f/src/clojure/zero_one/geni/defaults.clj#L6 is the "right thing" to do "in general", as there might always be Spark environments...

on my fork I made a change which fixes this: https://github.com/behrica/geni/commit/16080dd0156da170b301984ce79ff6ca699fc084

I have as well the impression, that even "requiring" the various geni namespaces, creates a session at one point in time. I saw it, because in a databricks cluster the...

so basically I found no way to get a uberjar with geni executing on databricks. (only all removals of "geni requires" made the jar executable)

I finally patched https://github.com/zero-one-group/geni/blob/develop/src/clojure/zero_one/geni/defaults.clj to have an empty map for session-config. This (I belive) still creates a session during "require" (which in my opinon is wrong), but at at least...

I don't remember any more about the details of the code changes. The "goal" is still to be able later to implement this: ``` (require-r-package '[datasets]) (r->clj r.datasets/iris {:as-tensor true})...