Integrating jupyterlab extensions

[costrouc]

I'm going to spend some time in the next few weeks to see how I can get a pure nix derivation for extensions. I will focus on the following extensions:

• [ ] ipywidgets
• [ ] jupyterlab-git

[MMesch]

great! Let us know if you need help! Really looking forward to integrating this into Nixpkgs as welll ...

[teuffy]

I installed the extensions and changed the shell.nix in line with the readme page but I can't see the extension options and icon in jupyter lab instance. Any special command to run? I Looked at googled but could not find any help. here my shell.nix

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  ihaskell = jupyter.kernels.iHaskellWith {
    name = "haskell";
    packages = p: with p; [ hvega formatting hmatrix ];
  };

  ipython = jupyter.kernels.iPythonWith {
    name = "python";
    packages = p: with p; [ numpy pandas matplotlib seaborn opencv pytorch scipy scikitlearn xgboost tensorflow-tensorboard keras-applications mysql-connector google_api_python_client sklearn-deap ipywidgets pprintpp virtualenvwrapper beautifulsoup4 requests html5lib memory_profiler jupyter_console statsmodels google_cloud_bigquery quandl pillow ];
  };

  jupyterEnvironment =
   jupyter.jupyterlabWith {
    kernels = [ ihaskell ipython ];
    directory = ./jupyterlab;
  };
in
  jupyterEnvironment.env

[guaraqe]

Seems ok, as long as you generated the jupyterlab directory with the command with mentioned in the README.

[teuffy]

Seems ok, as long as you generated the jupyterlab directory with the command with mentioned in the README.

I did it and here my jupylab folder tree

jupyterlab
├── extensions
│   ├── jupyterlab_bokeh-0.6.3.tgz
│   └── jupyterlab-ihaskell-0.0.8.tgz
├── schemas
│   └── @jupyterlab
│       ├── application-extension
│       │   ├── package.json.orig
│       │   └── sidebar.json

Should I run a specific command when I start jupyter lab from the shell as I am still not able to see the extensions in the browser?

[guaraqe]

Do you have jupyterlab installed in another way in the system? If yes, can you try to run nix-shell --pure? It happened once that the system was using the system-installed jupyterlab instead of the one given by the shell.

[teuffy]

Do you have jupyterlab installed in another way in the system? If yes, can you try to run nix-shell --pure? It happened once that the system was using the system-installed jupyterlab instead of the one given by the shell.

• I tried nix-shell --pure but no change - still not seeing the extensions
• I installed jupyter, at user level, with nix-env but after I installed it with shell.nix to see if I could get the extensions. Without success yet.
• I am using nixos and no jupyter installed at root level (but python installed).

[MMesch]

@costrouc , I think we can get a useful pure jupyterlab derivation now because jupyterlab 1.2.2 comes with a built-in extension manager.

This means that we could preresolve jupyterlab with all big standard extensions. The user can then select and disable them if he wants. Such a distribution would be more like the ubuntu inkscape or gimp packages that already come with most extensions preinstalled.

What do you think?

[costrouc]

Also FYI there are a lot of people asking for a better mechanism for jupyterlab extension installation and I think that the jupyterlab maintainers hear it. So I think what you are suggesting is a great solution especially since changes are coming.

[ariutta]

node2nix could help address this issue. Have you explored using it?

[MMesch]

@ariutta , yes. But I think we would need yarn2nix for this. We have already tried it once with @Profpatsch and got quite far - that is we downloaded all packages. But then we failed putting webpack and the whole build process into place. We stopped not because of this but because we weren't sure how to make this setup flexible enough for the user to select extensions himself.

[ariutta]

@MMesch, do we have examples of use cases? Would it be just allowing the user to select any combination of kernel(s) and JupyterLab extension(s)? What about nbextensions, Notebook server extensions and/or companion packages?

Are the following two examples representative of the use cases you'd like to handle?

1. Bob wants to use Jupyterlab with IPython, ipywidgets and jupyterlab-git
2. Susan wants to use Jupyterlab with IPython, Juniper, jupyterlab-vim, ipython-sql and jupyterlab-sql

[MMesch]

yes. I would add:

3. Herbert wants to add his own custom extension to the existing ones (this is a difficult one because we can't preresolve it)

We didn't focus much on server extensions for now, do you have an idea how to integrate them? I think they are basically python modules right? So maybe it is much easier to add them than the extensions that come via npm and yarn.

There is also another use case, more related tot the kernels, which would be nice to support:

• ideally a kernel environment can also be accessed without jupyter - e.g. via plain python which right now links to the jupyter lab python.
• ideally a package environment, e.g. for python or jupyter can easily be fed into the jupyterlab derivation

This is already possible but we can definitely make this experience a lot smoother

[ariutta]

Yes, I think you're right about server extensions.

ideally a package environment, e.g. for python or jupyter can easily be fed into the jupyterlab derivation

Would an example of this be using python packages from pypi2nix? The output of pypi2nix doesn't re-use existing python packages in nixpkgs.

[ariutta]

@MMesch, can you take a look at my fork? The changes are intended to support server extensions and to keep the jupyter directories isolated from each other. (Not ready for a pull request yet -- just a proof of concept for initial feedback.)

Here's an example of using my fork. Clone my fork of jupyterWith and the example repo to sibling directories to try it out.

Right now, jupyterWith shares the config, data and runtime directories across nix-shell environments. Only the directory for jupyterlab extensions is set up so it can be isolated (via the directory attribute for jupyterlabWith). In my PoC, I moved the directory attribute up and redefined it to be the parent of the jupyterlab extensions directory (now named lab) and the other directories config, data and runtime. I had to move it in order to use it here and in shellHook. Now shellHook creates this directory, if it doesn't exist, so the user doesn't have to worry about creating it.

I also allow the user to specify server extensions like this. I'd rather allow the user to pass in packages like this, but that didn't work when I tried it.

This setup isolates the various jupyter directories from environment to environment, but it doesn't yet make them immutable. That will be a little trickier to handle. Because the directories aren't immutable, users can use the normal CLI commands like jupyter-labextension install. I did need to use a few kludges to handle a jupyter bug, as you can see in install-extensions, to install a serverextension and two labextensions.

[ariutta]

I made additional proof of concept changes in my proposals branch, now supporting serverextensions, magics and lab extensions. You try it out with my demo repo.

[guaraqe]

I took a look at your branch, it seems that is suffers from the same indeterminism problems that our current branch has, right? In the sense that the versions of extensions are decided at the moment of build, so it can change over time. Just to be sure I understood.

[ariutta]

suffers from the same indeterminism problems that our current branch has, right?

Yes, that is correct. My branch is just an incremental change that does the following:

• sets the paths from jupyter --paths to isolate different jupyter projects from each other
• supports jupyter server extensions

Before addressing indeterminism, I want to first decide which jupyter paths should be isolated and which (if any) should be shared across jupyter projects.

If these changes are OK for now, I can make a pull request. Then in a future pull request, I can work on addressing indeterminism.

[guaraqe]

@MMesch I am not as familiar with Jupyter as you, so I don't know much about server extensions. What do you think? I have the impression that this just give us more features.

[MMesch]

@ariutta , this looks great! Do you think you could you open a PR for it?

[jamesdbrock]

There is ambition to rework the entension system in JupyterLab 3.0.

https://github.com/jupyterlab/jupyterlab/issues/8038

We want major features in 3.0 to entice users to upgrade. Some possibilities include:

• Extension system rework, allowing extensions to be installed without rebuilding JupyterLab https://github.com/jupyterlab/jupyterlab/issues/5672 https://github.com/jupyterlab/jupyterlab/issues/7468

[guaraqe]

@jamesdbrock This is very interesting for us, thank you for the information!

[jamesdbrock]

See also https://github.com/jupyterlab/jupyterlab/issues/7525

[jamesdbrock]

JupyterLab 3.0 is here, with the new extension build system, which I don't yet understand.

https://jupyterlab.readthedocs.io/en/latest/getting_started/changelog.html#v3-0

In JupyterLab 3.0, a new recommended way of distributing and installing extensions as Python pip or conda packages is available. Installing such extensions does not require rebuilding JupyterLab and does not require having NodeJS installed.

[jamesdbrock]

Here's a good issue to watch https://github.com/jupyterlab/jupyterlab/issues/9118

[guaraqe]

Thanks for the pointer, that can be really useful!

[ariutta]

I came up with a proof of concept for using the JupyterLab 3 prebuilt extensions from PyPi as well as prebuilding a JupyterLab extension for Nix. Here is where I generated Nix expressions for my NPM and PyPI dependencies. And here is where I specified how to build them all for inclusion in my shell.nix.

This isn't ready for a pull request, but I wanted to share it in case anyone else might find it useful.

[garbas]

This work is really interesting. We are currently working on some cleanup for jupyterWith (see main branch), but we have on our roadmap to package jupyterlab extensions the proper nix way.