Jupyter notebook cannot connect to pyspark kernel

[mreid-moz]

I launched two clusters yesterday, and neither was able to successfully run a notebook.

When opening a notebook, both stuck in the "Kernel starting, please wait..." state and could not run even simple python commands from the notebook. Rendering markdown cells worked as expected.

Running python manually from the commandline (via /mnt/anaconda2/bin/python) worked fine, as did running pyspark shell manually. A Zeppelin notebook was also able to run spark tasks as expected.

Output from the browser's js console:

 Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead[Learn More]  main.min.js:84
 Use of getPreventDefault() is deprecated.  Use defaultPrevented instead.  main.min.js:88:12747
 loaded custom.js  TestSerde.ipynb:121:15
 Default extension for cell metadata editing loaded.  main.min.js:24136:9
 Raw Cell Format toolbar preset loaded.  main.min.js:24222:5
 Slideshow extension for metadata editing loaded.  main.min.js:24270:9
 Use of Mutation Events is deprecated. Use MutationObserver instead.  main.min.js:88:7259
 actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...  main.min.js:29058:17
 load_extensions Arguments { 0: "nb_anacondacloud/main", 1: "nb_conda/main", 2: "jupyter-js-widgets/extension", 3: "jupyter-notebook-gist/extension", 4: "nbpresent/js/nbpresent.min", 5: "jupyter-spark/extension" }  main.min.js:12154:9
 Loading extension: nbpresent/js/nbpresent.min  main.min.js:12131:21
 Loading extension: nb_anacondacloud/main  main.min.js:12131:21
 Session: kernel_created (5f9f5c13-3017-4c95-a49a-dea071e3ef23)  main.min.js:23424:13
 Starting WebSockets: ws://localhost:8888/api/kernels/cd68a719-0de6-4f72-a0a4-03faa7629138  main.min.js:22710:9
 SecurityError: The operation is insecure.  main.min.js:22712
	Kernel.prototype.start_channels http://localhost:8888/static/notebook/js/main.min.js:22712:19
	Kernel.prototype._kernel_created http://localhost:8888/static/notebook/js/main.min.js:22659:9
	on_success http://localhost:8888/static/notebook/js/main.min.js:23479:13
	Session.prototype._on_success/< http://localhost:8888/static/notebook/js/main.min.js:23656:17
	l http://localhost:8888/static/notebook/js/main.min.js:87:24877
	fireWith http://localhost:8888/static/notebook/js/main.min.js:87:25702
	k http://localhost:8888/static/notebook/js/main.min.js:89:5346
	t/< http://localhost:8888/static/notebook/js/main.min.js:89:9152
 Loading extension: jupyter-js-widgets/extension  main.min.js:12131:21
 Loading extension: jupyter-spark/extension  main.min.js:12131:21
 Loading extension: jupyter-notebook-gist/extension  main.min.js:12131:21
 Loading extension: nb_conda/main  main.min.js:12131:21
 loaded widgets  extension.js:134:14

[robhudson]

Which EMR version did you choose when creating the cluster?

[mreid-moz]

This was using the default EMR version, which I believe was 5.2.1 at the time.

[washort]

@mreid-moz Is this issue still occurring for you?

[mreid-moz]

Yup, still happening for me (happens with the 5.8 EMR release too)