500 Error when Changing Kernel through Session

[bashterm]

The kernel gateway returns a 500 error when I try to change the Kernel in the current session.

I'm running the kernel gateway on docker.

Here is the server log:

jupyter_1   | [KernelGatewayApp] Starting buffering for c0169618-1f04-49fe-8b98-311f6df27668:270b28ea405236eaf399c29bf4be76e6
jupyter_1   | [E 180726 19:12:54 web:1621] Uncaught exception OPTIONS /api/sessions/df4015c9-9824-4c6f-854b-fe8601f0b148?1532632374048 (172.18.0.1)
jupyter_1   |     HTTPServerRequest(protocol='http', host='localhost:8888', method='OPTIONS', uri='/api/sessions/df4015c9-9824-4c6f-854b-fe8601f0b148?1532632374048', version='HTTP/1.1', remote_ip='172.18.0.1')
jupyter_1   |     Traceback (most recent call last):
jupyter_1   |       File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 1541, in _execute
jupyter_1   |         result = method(*self.path_args, **self.path_kwargs)
jupyter_1   |     TypeError: options() got an unexpected keyword argument 'session_id'
jupyter_1   | [KernelGatewayApp] WARNING | Unhandled error
jupyter_1   | [E 180726 19:12:54 web:2106] 500 OPTIONS /api/sessions/df4015c9-9824-4c6f-854b-fe8601f0b148?1532632374048 (172.18.0.1) 2.04ms

Any help or advice would be appreciated.

[kevin-bates]

I believe more information is required here. What does your request look like? I'm curious where the id df4015c9-9824-4c6f-854b-fe8601f0b148?1532632374048 is coming from since just prior to that, we see c0169618-1f04-49fe-8b98-311f6df27668:270b28ea405236eaf399c29bf4be76e6, the first portion of which presumably refers to the original kernel_id.

When I try changing kernels from within a Notebook configured with NB2KG to Enterprise Gateway (which should be the same behavior), I can successfully do that. However, the REST calls I see are all against /api/kernels, not /api/sessions. In addition, the sequence is to delete the original kernel, then create a new kernel (with a new id) via DELETE and POST calls respectively.

Based on your previous issue, I suspect you're attempting to implement kernel switches yourself. Have you studied how Notebook does this? I'm not familiar with pure Notebook operations, but I suspect observations using NB2KG would be the same since NB2KG is really just a kernel manager plugin. As a result, you might try issuing a similar sequence using /api/kernels.

I would also suggest setting up a parallel environment with Notebook and the NB2KG extension. You can find a docker image here. Although its geared toward Enterprise Gateway, there's no reason it wouldn't just work against JKG. Once this is setup, you can use it as a datapoint for comparing against what you're attempting outside of Notebook.

[bashterm]

I've been using the @jupyterlab/services javascript library to connect to the Kernel Gateway. The library implements the ability to change kernels through the session using a function they call changeKernel(). I'm not very clear on how the library itself works, because I don't know javascript very well.

I guess I don't fully understand the intended usecase is for the session api versus the kernel api?

It's possible I'm getting things wrong conceptually.

[kevin-bates]

Yeah, I agree with the confusion between session and kernel apis. I do know that creating a session will lead to the creation of a kernel.

I hoping some of the Kernel Gateway folks chime in here (@parente, @rolweber), but my understanding is that Kernel Gateway's (and, by extension Enterprise Gateway) primary purpose is to support running the kernel closer to the source (i.e., Spark cluster, more performant resources, etc.) - so sessions are not really a focus. This is supported by the fact that NB2KG (the Notebook extension that forwards requests to JKG) only forwards the /api/kernels and /api/kernelspecs apis. As a result, sessions remain on the Notebook server, yet the creation of a session will trigger the (remoted) creation of the associated kernel.

I'm not really a web person, but it seems like the handlers in JKG would merely forward requests to the underlying Notebook code, so it strikes me as odd that use of /api/sessions produces an invalid request.

[parente]

The session implementation in the notebook server relies (relied?) on sqlite to store session information on disk. In KG, we implemented our own SessionManager that holds all session information in memory. The persistence of session info in a database didn't appear valuable to the use case since all of the in-memory the kernel state is lost whenever a KG shuts down anyway.

The 500 error you're seeing hints that the SessionManager API may have changed in the notebook server, and that the implementation here in KG needs to be updated to match.

[bashterm]

OK. Is it possible then that I'm better off just writing my own typescript library to interact with the KG? It seems like the problem is that the frontend library offered by @jupyterlab/services is not connecting with the kernel gateway properly, because it's looking for parts of the API that aren't there.

I'm not using the whole library anyways, I don't need things like autocompletion or any of the notebook features.

[parente]

If the KG is providing an incompatible API vs notebook, we should fix it here, not require you to create a non-standard client. It requires some debugging, since I'm going on a hunch that it's an API incompatibility causing the problem and not something else.

[kevin-bates]

@bashterm, I'm not claiming to have cycles to look at this, but it would be helpful if you could provide some insight into the change_kernel() method or, better yet, the set of requests being submitted to Kernel Gateway on behalf of change_kernel() so we have a chance to reproduce the issue. I have a hunch that the original kernel was terminated prior to the failure against /api/sessions - by virtue of there being different kernel ids in the log snippet - but that would be good to confirm.

Thank you.

[bashterm]

I'll see what I can do. I've been bogged down lately with some work but hopefully I can work on this friday.