Merging in Nixpkgs

[guaraqe]

This issue will track the progress of the work for merging in Nixpkgs, announced here. I will do this here so that there is less noise in the Nixpkgs repo.

[mboes]

Do we want to make jupyterWith a flake instead?

[guaraqe]

I think there is not much reason for JupyterWith to not be in Nixpkgs. We made it outside of Nixpkgs because of technical reasons in the beginning, but I think the idea is mature enough now.

Also, we want it to be updated at the same rate as other packages in Nixpkgs. And since these dependencies are many and are not controlled by us, it will be easier if updates in Nixpkgs include tests for JupyterWith. This decentralizes work and relieves us of the update burden for multiple ecosystems we don't know well.

I think a Flake would be more adapted for the kind of LTS distribution of Jupyter + Data Science packages proposed by @MMesch.

[guaraqe]

There are two main points to update in Nixpkgs:

• pkgs/applications/editors/jupyter, which is strictly simpler than JupyterWith. This will be expanded a lot. The main difficulties are going to be the definition of overrides in a transparent way.
• nixos/modules/services/development/jupyter, which defines the NixOS service. Everything here will have to be ported to the JupyterWith interface, but most of the code will stay the same.

[mboes]

Interested in @edolstra's perspective on this one. @edolstra see discussion above.

[guaraqe]

I have a version with both IPython and IHaskell working. Surprisingly, IHaskell was working in Nixpkgs, and I just had to mark it as not broken. I will proceed to the other kernels, and start thinking about the tests.

[guaraqe]

Five kernels are building normally, three of them not:

• iJavascript, seems to be a sandbox problem which we didn't see before
• Juniper, it seems that it is not in rPackages anymore
• Ansible, whose package does not build

I am ignoring Xeus for the moment, since it is much more complex than the others, and a problem in Nixpkgs by itself.

[leonardp]

@guaraqe i love to see efforts going the way of merging jupyterWith with nixpkgs.

has progress been made on this? is any help needed?

[evanjs]

Will #99 impact the addition of jupyterWith to the jupyter modules for NixOS? Or would this still happen, just with more stable dependencies in the form of a flake?

[guaraqe]

The idea of a flake is for it to be separated from Nixpkgs, so the ecosystem can be more decentralized. I still have to try to do it, to see if is a good match for JupyterWith. If it works well, there will be no merge, if it doesn't, I will try merging once more.

[evanjs]

Hrm, alright. Dependencies aside, I'm hoping there's a way to get a module for either jupyter lab or jupyterWith at some point. Much easier to manage long-running services that way, in my experience.

[guaraqe]

Flakes allow to expose NixOS modules, so this should be possible. A reference: https://www.tweag.io/blog/2020-07-31-nixos-flakes/

[costrouc]

I'm building a jupyterhub service for nixos and here are all the kernels I was able to get running without adding new packages https://github.com/NixOS/nixpkgs/pull/94783#issuecomment-670018572. I would love to help getting some of the kernels ported over. Specifically the ones that are not available at the moment are not checked. I will keep this up to date.

• [X] :: python
• [X] rust
• [X] ansible
• [X] bash
• [X] irkernel
• [X] ihaskell
• [x] c :: https://github.com/NixOS/nixpkgs/pull/94783
• [ ] gophernotes :: https://github.com/NixOS/nixpkgs/pull/94902
• [x] inix(nix-kernel) :: https://github.com/NixOS/nixpkgs/pull/94948
• [ ] ijavascript :: https://github.com/NixOS/nixpkgs/pull/94945
• [x] iruby :: https://github.com/NixOS/nixpkgs/pull/94947
• [ ] xeus-cling
• [ ] juniper

@tonyfast this should track which kernels are available in nix.

[evanjs]

Flakes allow to expose NixOS modules, so this should be possible. A reference: https://www.tweag.io/blog/2020-07-31-nixos-flakes/

Thanks! Been trying to wrap my head around flakes; I really need to actually read the blog posts and start playing with them.

@costrouc Thank you! I'm excited to see this added as well!

[devhell]

@costrouc I'm also massively excited for this, thank you!

[doronbehar]

Hey all.

In the last few days I tried to package a few jupyter packages in nixpkgs, and I managed to package, with working tests as of the moment:

• xeus
• xeus-python

I also had to package a few other dependencies of them as well. All work was done completely in nixpkgs at:

https://github.com/NixOS/nixpkgs/compare/master...doronbehar%3Apkg%2Fxeus

I started working on packaging those in nixpkgs with the aim to add support for xeus-python kernel here, but I eventually decided not to use Jupyter at all for the task I have at hand. That's why I haven't opened a PR but anyone else is free to take the lead from here.

I also added the jupyterlab-lsp package to nixpkgs, (see this), which might interest some of you.

[garbas]

We are converting current master into a flake and reworking the API. You can follow the progress in main branch. In the roadmap we have we also plan to expose NixOS module for jupyterhub.