Shift some features from kernel gateway into notebook server?

[SimonBiggs]

@parente I don't have a good understanding of the enormity of the task. I was wondering how plausible is it to merge the features provided by kernel gateway back into the notebook server? Essentially deprecating this code base and by providing "gateway" flags to the netbook server have it provide the necessary changes to become the kernel gateway equivalent.

[SimonBiggs]

I imagine the first step would be to copy all of the tests over, then to implement flags that switch the notebook server to a kernel gateway. Update the tests to use those flags. Then beginning pulling over missing features. CORS is the primary reason I myself am using kernel gateway. I don't have a good handle on the other extra features that kernel gateway provides.

[SimonBiggs]

I imagine the real first step is, if it is desired by you, would be to start the conversation over at the notebook server repo.

[SimonBiggs]

Or is the difference between using REST and websockets vs zeromq such a massive fork in the road that code base overlap in reality is actually minimal?

[parente]

@rgbkrk and I started a conversation about this topic at JupyterCon (read: talked about it for 30 seconds). A couple things to note:

• There is very little reimplementation of code here vs in the notebook server. The KG code simply imports various notebook packages, reuses them, and overrides them in places.
• The "notebook-http" mode supported by KG probably has no place in the notebook server. It has nothing to do with the Jupyter protocol or the notebook UI.

I struggle with whether the term "diverging" is applicable here. The situation more feels like Flask-RESTFul and Flask. The former is an extension of the latter for a specific purpose. I think that's a more apt description of KG and notebook server respectively, but I can surely be convinced otherwise.

Maybe a better question to start with is: is there value in integrating certain KG features directly into the jupyter/notebook project? One example: a kernel activity API was added to the notebook in jupyter/notebook#1827 and removed here in jupyter/kernel_gateway#230.

[SimonBiggs]

That makes sense. Thanks @parente.

[rolweber]

The configurable limit on the number of kernels is something that could be moved to juypter_client.

Something I like about kernel gateway in websocket mode vs. a notebook server is that it doesn't have a content manager. So if you're running a kernel gateway, it is clear that your server offers only kernels, but not notebooks. I guess it would be possible to disable the extra parts in a notebook server though.

[rgbkrk]

My immediate thought is that we should accept configurable options for the notebook server that make a deployment compatible with the kernel gateway. CORS is currently configurable. We could make a mode that disables all file serving and non kernel APIs --gateway-mode.

I recently talked to @lresende and others about a future direction of the kernel gateway that couldn't readily be done with the standalone notebook server -- it would be multi-tenant, and schedule kernels across a cluster.

[SimonBiggs]

@rgbkrk I wasn't able to get CORS to work on the notebook server without also having the options "allow_headers" and "allow_methods" that kernel gateway provides. I gave a bit more info over at https://github.com/jupyter/notebook/issues/2644

Not sure if I missed something though...

[rgbkrk]

I just answered in the other thread, so I think this means we'd need to replicate all the functionality from kernel gateway over. If you start making a list you can subdivide the work ahead. Right now it's looking like we'd want:

• [ ] --gateway-mode (disables file serving and non-kernel/kernelspec APIs)
• [ ] Explicit allow_headers and allow_methods (I think this gets buried away in Tornado settings at the moment)

[SimonBiggs]

@rgbkrk The meat of enabling allow_headers and allow_methods is within the CORSMixin class: https://github.com/jupyter/kernel_gateway/blob/fc1d714eecacacc4447ec516578d9032c1f2b574/kernel_gateway/mixins.py#L12-L43

Would you be okay with me attempting a pull request to include that class within the notebook server pulling it in where it is needed?

[SimonBiggs]

On second thoughts really there is probably a much more "notebook specific" way which essentially adds the following for allow_headers and allow_methods: https://github.com/jupyter/kernel_gateway/blob/fc1d714eecacacc4447ec516578d9032c1f2b574/kernel_gateway/mixins.py#L30-L33

I'll have a look more into the notebook codebase. Try and find my feet.

[SimonBiggs]

@rgbkrk Scrap that. Your suggestion worked. I had to turn off xsrf cookies, then using token access worked perfectly.

https://github.com/jupyter/notebook/issues/2644

[rgbkrk]

That's great!

[kevin-bates]

@SimonBiggs - Since the hood is up, I think this might be a good time to check out jupyter server if you haven't already.

[SimonBiggs]

@kevin-bates thank you for the heads up. Massively appreciated. I'll take it for a spin on the weekend hopefully.