'conf' is not defined for version 4.1.0

[rsdmse]

I am getting the following error for version 4.1.0 (jupyterlab 3.1.9, pyspark 3.1.2):

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_139706/319767949.py in <module>
      2 
      3 # start the spark context using the SparkConf the extension inserted
----> 4 sc=SparkContext.getOrCreate(conf=conf) #Start the spark context

NameError: name 'conf' is not defined

However, this works fine in the following scenarios:

• in your docker container (jupyterlab 3.0.16, pyspark 2.4.5, jupyterlab-sparkmonitor 3.0.1)
• in my own conda environment (jupyterlab 3.1.9, pyspark 2.4.8, jupyterlab-sparkmonitor 3.1.0)

I wonder if pyspark 3.1 is not supported yet?

[rsdmse]

I realized this could be due to the kernel.json file created by python -m ipykernel install ...:

{
 "argv": [
  "$HOME/envs/pyspark3.0/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "pyspark3.0",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

If I change it to:

{
 "argv": [
  "$HOME/.local/share/jupyter/kernels/pyspark3.0/init.sh",
  "-f",
  "{connection_file}"
 ],
 "display_name": "pyspark3.0",
 "language": "python"
}

where init.sh:

#!/bin/bash
source $HOME/envs/pyspark3.0/bin/activate
export IPYTHONDIR=$HOME/.ipython
python -m ipykernel "$@"

then it works. Maybe this is related to issue #24. However, nothing is shown in the "Spark UI" tab.

Using the same setup for pyspark 3.1, I got no errors but the live monitoring tool doesn't appear.

[rsdmse]

In the jupyter lab log:

[W 2021-09-01 06:59:35.270 ServerApp] 404 GET /4040 (127.0.0.1) 22.59ms referer=None

But sc.uiWebUrl shows the port number is 4040.