Luciano Resende

What is the difference with setting `HADOOP_CONF_DIR` in `$SPARK_HOME/spark-env.sh` ? More generically, why manage system wide configurations in the kernelspec?

Maybe I am misunderstanding this, while with vanilla Toree you have the option to get your own local spark started (e.g. local[...]), when considering an enterprise environment where there is...

@ribamar-santarosa Thanks for the explanation, I believe I understand the scenario now and have just minor comments on the changes.

@paf91 @patilsanket48 I haven't used 3.5.3 yet, but you should look into the spark driver logs and see what is causing the spark session creation to fail, probably some additional...

@patilsanket48 this is what controls the session initialization: https://github.com/jupyter-server/enterprise_gateway/blob/ffebbaff9184cc48e0c48c59682eac0a2fbf1c45/etc/kernelspecs/spark_python_kubernetes/kernel.json#L30 And this parameter is described in https://jupyter-enterprise-gateway.readthedocs.io/en/latest/contributors/system-architecture.html ``` Other options supported by launchers include: --RemoteProcessProxy.port-range {port_range} - passes configured port-range to...

Both ways should accomplish the same, the idea is that you can set a global/default one, and have the ability to overwrite on the kernel level. @jason810496, this looks good,...

@jason810496 could you please rebase the pr

#3245 should fix the doc build

https://elyra.readthedocs.io/en/latest/user_guide/env-variables-file-based-nodes.html

> conda create -y -n elyra-image-env python=3.8 --channel conda-forge should be python=3.9? And there are some config options for configuring conda-forge channels in https://mamba.readthedocs.io/en/latest/user_guide/troubleshooting.html#defaults-channels