SlicerJupyter icon indicating copy to clipboard operation
SlicerJupyter copied to clipboard

Kernel dying out repeatedly

Open tashrifbillah opened this issue 4 years ago • 25 comments

Followed your video, copied command to clipboard and installed on a Terminal that I launched from Notebook:

image

Installation went fine.

Error snapshot:

image

It's dying every three seconds or so.

@pieper might be able to reproduce this error.

Since our data are on the ERIS cluster, we won't be able to use public binder to run SlicerJupyter on. On the other hand, we won't have docker. So, we are left with making use of Parters Jupyter server to run SlicerJupyter on.

tashrifbillah avatar Jun 11 '20 22:06 tashrifbillah

Can you check if you can launch Slicer successfully on that system (without Jupyter) and maybe get a stack trace?

lassoan avatar Jun 11 '20 22:06 lassoan

Okay, it looks like it isn't even starting properly ... dying right after connecting.

tashrifbillah avatar Jun 11 '20 22:06 tashrifbillah

Most common issue in a server environment is wrong X or OpenGL configuration. See all the questions on the Slicer forum about running Slicer on a headless system.

lassoan avatar Jun 11 '20 22:06 lassoan

Can you check if you can launch Slicer successfully on that system

It does. image

running Slicer on a headless system.

I know about compatibility mode we discussed on Slicer forum. Even if that is the issue, which is not supported by the above log, how may that help me solving this issue?

tashrifbillah avatar Jun 11 '20 22:06 tashrifbillah

@pieper I guess your experience is needed here.

lassoan avatar Jun 11 '20 23:06 lassoan

@pieper I guess your experience is needed here.

Sorry, I've never used a Partners Jupyter server, but I probably have access if you need help debugging. @tashrifbillah can you describe exactly what system you are using?

pieper avatar Jun 12 '20 14:06 pieper

  • grx04.research.partners.org-- a CentOS 7 node through NoMachine
  • module load libGLU/9.0.0-foss-2018b
  • https://jupyterhub.partners.org

tashrifbillah avatar Jun 12 '20 14:06 tashrifbillah

@pieper if you please hold, I am looking at ~/.local/share/jupyter/kernels/slicer-4.11/kernel.json and might have found the problem.

tashrifbillah avatar Jun 12 '20 14:06 tashrifbillah

Never mind.

If I try the following on a Linux terminal:

[tb571@grx04 slicer-4.11]$ cat kernel.json
{
    "display_name": "Slicer 4.11",
    "language" : "python",
    "argv": [
        "/data/pnl/soft/pnlpipe3/Slicer-4.11.0-linux-amd64/bin/../SlicerWithExtensions.sh",
        "--no-splash",
        "--python-code",
        "connection_file=r'{connection_file}'; print('Jupyter connection file: ['+connection_file+']'); slicer.modules.jupyterkernel.startKernel(connection_file);slicer.util.mainWindow().showMinimized()"
    ]
}

I get this:

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-tb571'
Switch to module:  "Welcome"
Jupyter connection file: [{connection_file}]
startKernel connectionFile does not exist "{connection_file}"

Is this expected?

tashrifbillah avatar Jun 12 '20 15:06 tashrifbillah

Jupyter replaces {connection_file} in kernel.json with the actual connection file. You need to do the same if you want to test it by running the command yourself.

lassoan avatar Jun 12 '20 15:06 lassoan

Hi @pieper , did you get a chance to look at it? In the meantime, I found Partners VPN connection instruction which was easy to follow on my Windows laptop. After connecting through VPN, you can log into https://jupyterhub.partners.org/ from your laptop browser. Once you are in https://jupyterhub.partners.org/, you can open a terminal on the Notebook and follow SlicerJupyter installation instructions.

This is a way around NoMachine in case you don't have that setup. Happy to work with you over a call as well.

tashrifbillah avatar Jun 19 '20 14:06 tashrifbillah

Hi @lassoan , is there a log where I may find the events during kernel initiation and failure that I described in the above?

tashrifbillah avatar Jun 19 '20 14:06 tashrifbillah

Hi @tashrifbillah - I don't really use partner's jupyterhub or slicerjupyter much myself, so not sure I can do anything other than you will be able to do. Let's see what @lassoan suggests for debugging techniques and I can jump in if needed.

pieper avatar Jun 19 '20 16:06 pieper

Slicer logs everything into the Slicer log files.

lassoan avatar Jun 19 '20 21:06 lassoan

I confirm that it is an OpenGL issue. https://jupyterhub.partners.org/ X environment is probably not set up properly.

tashrifbillah avatar Jun 19 '20 22:06 tashrifbillah

Hi,

I am still trying to find a way out of this problem.

The following pops up Slicer-4.11 GUI and stays there:

(base) [tb571@grx05 slicer-4.11]$ jupyter console --kernel slicer-4.11
Switch to module:  "Welcome"
Jupyter connection file: [/PHShome/tb571/.local/share/jupyter/runtime/kernel-55305.json]
Run with XEUS 0.23.13
Starting Jupyter kernel server
Jupyter console 6.1.0

  __  _____ _   _ ___
  \ \/ / _ \ | | / __|
   >  <  __/ |_| \__ \
  /_/\_\___|\__,_|___/

  xeus-python: a Jupyter kernel for Python
  Python 3.6.7
In [1]:                                                                                

Is there a way to suppress that?

tashrifbillah avatar Jun 30 '20 21:06 tashrifbillah

The following pops up Slicer-4.11 GUI and stays there

The same happens if I open the kernel from http://localhost:8889/tree .

tashrifbillah avatar Jun 30 '20 22:06 tashrifbillah

Can you post a screenshot? Is the problem that the text is displayed or that something does not work correctly?

lassoan avatar Jul 01 '20 02:07 lassoan

The ascii word art is displayed the Jupyter client calls kernel_info_request. I don't remember any of the Jupyter clients calling this or at least it did not display the result. What Jupyter client do you use?

lassoan avatar Jul 01 '20 02:07 lassoan

What Jupyter client do you use?

(base) [tb571@grx05 slicer-4.11]$ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.0.3
qtconsole        : 4.7.4
ipython          : 7.8.0
ipykernel        : 5.3.0
jupyter client   : 6.1.3
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 5.0.7
traitlets        : 4.3.2


tashrifbillah avatar Jul 01 '20 02:07 tashrifbillah

I guess my question is about the default behavior-- if I open a kernel, should Slicer GUI be open and stay minimized in the taskbar?

Are these the screenshots you asked for?

image

Minimized Slicer-4.11 on the taskbar


image

Maximized the minimized Slicer-4.11


tashrifbillah avatar Jul 01 '20 02:07 tashrifbillah

Is the problem that the text is displayed or that something does not work correctly?

The problem is, we do not have any GUI access to the server other than Jupyter notebook where Slicer-4.11 Kernel would be running. And coming back to the issue I opened, the kernel fails to start attempting to launch Slicer-4.11 and failing.

tashrifbillah avatar Jul 01 '20 02:07 tashrifbillah

Well, I think that is the default behavior as specified by "slicer.util.mainWindow().showMinimized()":

  "/PHShome/tb571/tmp/Slicer-4.11.0-linux-amd64/Slicer",
  "--no-splash",
  "--python-code",
  "connection_file=r'{connection_file}'; print('Jupyter connection file: ['+connection_file+']'); slicer.modules.jupyterkernel.startKernel(connection_file);slicer.util.mainWindow().showMinimized()"

tashrifbillah avatar Jul 01 '20 03:07 tashrifbillah

I don't remember ever seeing this XEUS graphics, but Slicer starting up minimized is the expected behavior. You don't need GUI access to the server, you should be able to do visualizations, processing, etc. all just using the Jupyter interface. The server has to provide minimal graphics resources, but if even the free binder service can meet these requirements it is hard to imagine how a dedicated, more powerful server would have any problem with this.

lassoan avatar Jul 01 '20 03:07 lassoan

@tashrifbillah are you able to run Slicer on the remote server using Xvfb or Xdummy?

pieper avatar Jul 01 '20 12:07 pieper