Flakes UX: make it easy to evaluate an arbitrary expression in a context of a flake

[balsoft]

This is a design discussion issue, so feel free to give as much feedback and criticism as you like.

Preamble

In ye olde days of nix-shell, it was possible to easily evaluate expressions in the context of nixpkgs. Something like this:

nix-shell -p 'python3.withPackages (ps: with ps; [ numpy scipy ])'

Drops you into a shell in which there's a python3 executable with numpy and scipy available. This is really useful for ad-hoc scripting. You could also do something like

nix-shell -p 'hello.overrideAttrs (_: { src = ./.; })'

It was also possible to add alias p='nix-shell -p' to save some typing.

The flexibility is endless -- and all just a few keystrokes away!

Current situation

It is possible to do something similar with new, flaky interface:

nix shell --impure --expr 'with builtins.getFlake "nixpkgs"; with legacyPackages.${builtins.currentPlatform}; python3.withPackages (ps: with ps; [ numpy scipy ])'

However, this is a chore to do, and I'd much rather just run pip install numpy scipy and deal with all the usual stateful package management issues than type this monstrosity every time I need to try a python package.

Proposal

Add a way to easily evaluate expressions "in the context of" a flake, meaning with the top-level attributes and {packages,legacyPackages}.${currentPlatform} of that flake available in scope.

Possible implementations

I came up with three different ideas, all of which have some advantages and drawbacks.

(my current favourite) Allow flake fragments to be arbitrary expressions

Example: nix shell 'nixpkgs#(python3.withPackages (ps: with ps; [ numpy scipy ]))'

This feels like a natural extension of the already-existing flake-fragment-as-attrpath interpretation. It shouldn't be too difficult to implement, and doesn't break compatibility. It does however introduce some potential complexity to newcomers.

With alias p='nix shell' it is possible to do p 'nixpkgs#python3.withPackages (...)' , and some more keystrokes could be saved with a registry entry n -> nixpkgs: p 'n#python3.withPackages (...)'

Add a new flag, like --with

Example: nix shell --with nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

This plays nicely with Nix' with keyword. It implies that some expression will be evaluated with some additional context.

Aliasing p='nix shell --with nixpkgs' can give experience identical to the status quo -- p 'python3.withPackages (...)'

Change the behaviour of --expr

Example: nix shell --expr nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

This breaks compatibility, but may be nicer than --with since --expr was not very usable with flakes anyways and it's not nice to have a lot of useless flags around.

[Radvendii]

+1 to nix shell nixpkgs#(<expression>) syntax

One thing I wonder about also is being able to pull multiple flakes into scope. (the equivalent of with x; with y;) Maybe with a syntax like [nixpkgs otherFlake]#(<expression>) Maybe once you get to this point it's better to make a flake.nix file and take them both in as inputs.

This may go too far down the road of complexity without enough real use-cases.I remember there was a use-case for this that came up for me a little while ago, but I don't remember what it was. I'll write it down if it ever comes up again.

EDIT: I talked it through with @balsoft and we're in agreement that my addition isn't a great move right now. It changes the semantics of flake paths away from being "urls" of sorts, which is a much bigger decision, even if we want to go there.

[thufschmitt]

I’m a big fan of the overall idea, that’s something I want to have available reasonably frequently, and it’s always a pain. I’m not sure what the UX should be though:

• The nixpkgs#(expression) syntax would make a lot of sense if the Nix language had a x.(expr) syntax (like what OCaml has for modules), but it doesnt, (and for good reasons imho). So it’s a bit confusing. (and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs)
• --with <flakeref> sounds nice, but since it only makes sense with --expr, which isn’t very clean
• --expr <flakeref> <expr> isn’t really viable, because part of the point of --expr is to make it possible to use the nix command with non-flake things.

A fourth way could be to have a --expr-with <flakeref> <expr> flag. But it’s not utterly pretty either.

[balsoft]

My --with flag suggestion is actually --with <flakeref> <expr>, so like your --expr-with suggestion. But having --with <flakeref> --expr <expr> is also a possible solution.

The nixpkgs#(expression) syntax would make a lot of sense if the Nix language had a x.(expr) syntax

I think the installables syntax is already different from usual Nix syntax. E.g. <flakeref>#<attrpath> will not evaluate to the same thing in actual Nix language as it does in the attributes, so I think some syntactic liberty could be allowed. But I also see your point, because it can be even more confusing for newcomers if there's even more strange syntax to learn and understand.

[Radvendii]

and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs

As far as I understand it, this evaluates to the same thing in the existing and proposed cases. It's like the difference between (with foo; bar) and (foo.bar)

[thufschmitt]

and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs

As far as I understand it, this evaluates to the same thing in the existing and proposed cases. It's like the difference between (with foo; bar) and (foo.bar)

Not exactly. Current nixpkgs#(foo) is (builtins.getFlake "nixpkgs").[Search path Magic]."(foo)" while with this proposal it would be (builtins.getFlake "nixpkgs").[Search path Magic].foo (so the attr name is different). But that’s really nitpicking because that’s a really small edge-case (and the syntax is officially unstable anyways)

[edolstra]

For the new UI, we have to careful about adding features that are not discoverable or have a steep learning curve. For instance, in the example

nix shell --expr nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

it's not obvious how a user would discover the existence of python3.withPackages or figure out that they need to write (ps: ...).

There's also the issue of cacheability. Currently we can cache flake output attributes like nixpkgs#pythonPackages.foo, but not arbitrary expressions like the above.

[zimbatm]

If we can translate CLI arguments to function calls, it's possible to build functions that work for common cases. This assumes some cooperation between nixpkgs and the nix CLI.

Eg: nix apply nixpkgs#python3.with --packages numpy,scipy

Could map to (builtins.getFlake "nixpkgs).legacyPackages.${builtins.currentSystem}.python3.with { packages = "numpy,scipy"; }

It makes me think of httpie since they also try to convert CLI to JSON data structures: https://httpie.io/docs#non-string-json-fields

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-22/16251/1

[bmabsout]

I recently wanted to do the exact same thing as @balsoft, and I did the exact same workaround. I didn't know whether it was supported natively or not so here's what I tried (What I intuitively thought would work): nix shell nixpkgs/nixos-21.11#python39 --expr "withPackages (p: [p.numpy])" I expected python39's attributes to be in scope for the expression but it wasn't

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-shell-for-python-packages/16575/4

[stale[bot]]

I marked this as stale due to inactivity. → More info

[K900]

Bad bot.

[Minion3665]

I recently wanted to do the exact same thing as @balsoft, and I did the exact same workaround. I didn't know whether it was supported natively or not so here's what I tried (What I intuitively thought would work): nix shell nixpkgs/nixos-21.11#python39 --expr "withPackages (p: [p.numpy])" I expected python39's attributes to be in scope for the expression but it wasn't

I wanted this to work too, so I made a patch for it: https://github.com/Minion3665/nix/compare/master...Minion3665:nix:5567-make-installables-expr-context.patch

Right now it only works on nixos-unstable (I believe it needs nix 2.9? I've only properly tested it on 2.8 (failing) and 2.10 (working fine) though)

You can apply it by doing something like this in an overlay:

final: prev:  {
  nixFlakes = prev.nixFlakes.overrideAttrs (old: {
    patches = (old.patches or []) ++ [
      ./nix/5567-make-installables-expr-context.patch
    ];
  });
}

where ./nix/5567-make-installables-expr-context.patch is the path to the patch file.

The patch works by adding with statements to the front of your expression; in the future I hope to improve the patch by doing something similar to how nix repl's :l command works but for now I wanted something that was a little quicker for me to make.

[GuillaumeDesforges]

@Minion3665 please upstream this patch, it's awesome :star_struck:

[Uthar]

I wanted this as well. I created this shell function until it's resolved:

# Python with packages
pwp () {
    local python=$1;
    shift;
    local pkgs=$@;
    nix build \
        --impure \
        --expr "with builtins; with getFlake \"nixpkgs\"; (getAttr currentSystem legacyPackages).$python.withPackages (ps: with ps; [ $pkgs ])"    
}

Example:

pwp python310 pika boto3 kafka-python
result/bin/python

It's even better than the nix-shell -p stuff IMHO

[YorikSar]

If Nix was to allow specifying functions in place of "installables" and passing positional arguments to them, we could use it like: nix shell nixpkgs#python3.withPackages --argPos '(ps: with ps; [ foo bar baz ])'. That said, Nix doesn't support passing positional arguments at all, and probably for a good reason.

[viperML]

nix eval has a similar flag, --apply, that maybe could be ported to nix shell/build

$ nix eval --apply "p: (p.withPackages (pp: [pp.requests])).outPath" nixpkgs#python3
"/nix/store/vfyr42phq4h4gz63zg9qizz61a59503r-python3-3.10.8-env"

[bobvanderlinden]

I came here by searching "nix shell flake expr" and my use-case was python3.withPackages as well! 😅

My reasoning was that since the following works with the repl:

$ cd ~/projects/nixpkgs
$ nix repl .
nix-repl> python3.withPackages (pkgs: [ pkgs.pjsua2 ])
«derivation /nix/store/kchdd811lan46mlk8bdhqq70i87lv2r6-python3-3.10.9-env.drv»
nix-repl> :b python3.withPackages (pkgs: [ pkgs.pjsua2 ])

this derivation produced the following outputs:
  out -> /nix/store/cyc8nhbf0w4dsydi1alvby4xgcg3zja5-python3-3.10.9-env

Then why doesn't this work for nix shell?:

$ nix shell . --expr 'python3.withPackages (pkgs: [ pkgs.pjsua2 ])'

That doesn't make sense to me.

The patch of @Minion3665 seem to align with this reasoning. I couldn't find a PR of your branch, is there a PR already?

If not, do you mind to create one? Or if it helps, is it ok for me to create one from your patch?

[Minion3665]

That doesn't make sense to me.

The patch of @Minion3665 seem to align with this reasoning. I couldn't find a PR of your branch, is there a PR already?

If not, do you mind to create one? Or if it helps, is it ok for me to create one from your patch?

There is no pr yet, I've considered making one however:

• I'm pretty sure the patch doesn't work on the latest master, some work would need to be put in to make it work again
• The patch tacks on a with statement to the given expression and calls it a day, this is both a little janky and also can cause error messages to be obscured by a blob of 'with' garbage

For these reasons I haven't PRed, however I'm more than willing to have someone else make a PR or to work with someone on improving the patch and PR myself.

I've planned to do these and PR for a while but I'm sure you know how life can be...

[n8henrie]

Thanks to posters above for giving me a clue how to access e.g. pkgs for interactively running tests:

$ nix eval --impure \
    --apply 'pkgs: pkgs.callPackage pkgs/development/libraries/science/math/libtorch/test { cudaSupport = false; }' \
    .#legacyPackages.aarch64-darwin

Something like this for nix shell would be amazing (again, with the python + packages example being what has brought me to this thread most frequently).

[n8henrie]

Adding on to examples like @viperML from above, I hadn't realized that nix shell accepts a path, so one can combine nix eval --raw --apply with nix shell and get a reasonable (?) replacement for nix-shell:

For example, working on a local checkout of nixpkgs:

$ nix shell "$(
        nix eval --raw --apply '
            py: (py.withPackages (pp: [ pp.torch ]))
        ' .#python310
    )" \
    -c python -c 'import torch; print(torch.__version__)'
2.0.1

Interestingly, I was initially getting segfaults and stack overflows trying to get this to work (maybe this issue?):

$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
Segmentation fault: 11
$
$ # on another machine:
$ nix eval --apply 'py: (py.withPackages (pp: [ pp.requests ]))' nixpkgs#python310
...
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
error: stack overflow (possible infinite recursion)

But simply adding .outPath made it work (thanks @viperML):

$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ])).outPath' nixpkgs#python310
"/nix/store/ir5m4j69icbgh7rgi8hpcn75jl16jshq-python3-3.10.12-env"
$ nix eval --apply 'py: (py.withPackages (pp: [ pp.requests ])).outPath' nixpkgs#python310
"/nix/store/89smqkd3q9vqfjbxv12hlaznp5v1j39i-python3-3.10.12-env"

So does --raw:

$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
Segmentation fault: 11
$ nix eval --raw --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
/nix/store/ir5m4j69icbgh7rgi8hpcn75jl16jshq-python3-3.10.12-env
$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
Segmentation fault: 11

--raw is required to strip the extra quotes from the path anyway, so it seems reasonable to use.

Oddly, when I add a second package it fails:

$ nix build "$(nix eval --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ]))' nixpkgs#python3)"
error: path '/nix/store/h2i9izgfc5fh5c4b4nrg1pjn5y12axzw-python3-3.10.12-env' is required, but there is no substituter that can build it
$ nix-shell -I nixpkgs=~/git/nixpkgs -p 'python310.withPackages (ps: with ps; [ flask requests ])'
[nix-shell:/tmp/test]$ echo $?
0

[erikarvstedt]

@n8henrie, here's a fix for your last cmd:

nix shell "$(nix eval nixpkgs#python3 --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).drvPath')"

In general (and as you mentioned) you can run nix on arbitrary drv expressions that have single flake output as their input like so:

nix <cmd> "$(nix eval nixpkgs#pkgs --raw --apply 'pkgs:
 (<arbitrary expression that yields a derivation>).drvPath
')"

[n8henrie]

Thanks! This also seems to work pretty well and doesn't require a separate call to nix shell:

"$(nix eval --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).out' .#python310)"/bin/python -c 'import flask; print(flask.__version__)'

[n8henrie]

A bash function similar to what @Uthar uses above:

$ nixShellWithPythonPackages() {
    nix shell "$(nix eval --raw --apply "py: (py.withPackages (pp: with pp; [ $* ])).drvPath" nixpkgs#python310)"
}
$ nixShellWithPythonPackages torch requests
$ type -p python
/nix/store/p306n49i437di59p6lwn33qcksqrjk3z-python3-3.10.12-env/bin/python
$ python -c 'import torch; import requests; print(torch.__version__, requests.__version__)'
2.0.1 2.29.0

[erikarvstedt]

This also seems to work pretty well and doesn't require a separate call to nix shell:

This only works because the drv has already been built previously. You want this:

$(nix build  --no-link --print-out-paths "$(nix eval nixpkgs#python3 --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).drvPath')")/bin/python -c 'import flask; print(flask.__version__)'

[xaverdh]

you probably want to use the nix shell variant to make sure the path isn't garbage collected between the call to nix eval and the execution of the python binary

[YellowOnion]

I second any sort of higher order implementation.

Infact I think we should match flake.nix output syntax:

$ nix build nixpkgs# github:somerepo#pkg --with '{nixpkgs, pkg}: nixpkgs.pkg2.override {inherit pkg;}'

[iFreilicht]

I'm interested in implementing this. It looks like a high-demand feature that could bridge the gap between the simple nix shell nixpkgs#asdf commands and writing a full-on flake. Just to make sure I'm grasping the problem and solution space correctly, here are my thoughts and the two potential solutions:

--apply, solution with precedent

The least controversial (but also least "nice") solution would be to port the --apply flag from nix eval to nix shell and nix build (or to any command that takes installables in general):

$ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))'

Differences to nix eval

This solution should not require the .outPath workaround you need for nix eval, it should not require --impure because the function itself is pure, and it should work for multiple installables, but nix eval applies the function passed to --apply to all of its inputs, which will very rarely be useful here. Rather, we want to be able to apply different functions to different installables.

There are two potential solutions for this:

Flake-like outputs functions

This is the option proposed by @YellowOnion:

$ nix shell nixpkgs github:somerepo#pkg --apply '{nixpkgs, pkg}: nixpkgs.pkg2.override {inherit pkg;}'

Unused inputs would just be put into the shell without applying the function:

$ nix shell nixpkgs#python3 nixpkgs#gnugrep --apply '{python3}: (python3.withPackages (p: [ p.flask ]))'

This is cool because it is very similar to how flakes work, making the transition easy and giving users a familiar interface. However, you have to type the name of the input three or more times, which is very annoying.

Split options

Basically the way --apply currently works, but extended to an arbitrary number of functions and installables.

$ nix shell nixpkgs github:somerepo#pkg --apply 'pkgs: p: pkgs.pkg2.override {pkg: p;}'

This is nice because it can be much terser. Again, if some inputs are unused, they will just be used verbatim.

$ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))' nixpkgs#gnugrep 

Note that this order is not mandatory, but good style as it puts the function close to its input. It's also important to note that now the evaluation of the functions depends on the order of the installables. This is not the case with the flake-like solution.

Closer to nix repl: --expr

I do like the idea from @bobvanderlinden as well to align more with the interface of nix repl:

$ nix repl .
nix-repl> python3.withPackages (pkgs: [ pkgs.pjsua2 ])
«derivation /nix/store/kchdd811lan46mlk8bdhqq70i87lv2r6-python3-3.10.9-env.drv»
$ nix shell . --expr 'python3.withPackages (pkgs: [ pkgs.pjsua2 ])'
# or
$ nix shell nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])'

Very clean, perfect for this simple usecase. I can't think of a way to really make this work with multiple installables, though.

Additionally, this doesn't cover all use-cases, it just makes the simple ones even simpler and allows jumping directly from a working repl statement to a nix shell invocation. So if this was added, I think it would have to be done in addition to --apply, not as a replacement.

Collision with current --expr installables

This solution completely reverses the current meaning of --expr, which replaces the initial part of the installable:

#           <installable> <                    output                         >
$ nix shell       nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])'
#           <        installable       >  <output>
$ nix shell --expr 'import <nixpkgs> {}'  hello

Alternative 1: just call it --eval

Somewhat arbitrary, just call it --eval instead of --expr. Not a great solution, but not terrible either.

Alternative 2: two argument flag --with

A better option might be to implement this as 2-argument flag --with that can mirror the behavior of with inside the nix language:

$ nix shell nixpkgs#gnugrep --with nixpkgs#python3 'withPackages (p: [ p.flask ])'  

This makes it very clear which function is applied to which flake, and makes confusing ordering an explicit error.

Interop

With other installables

Apart from flakes outputs, we also have store paths, files, and (as already mentioned) expressions.

For store paths, this feature doesn't seem to be applicable. For files, the --apply and --eval functions could operate on the output of those files. Only the proposed --with would not be compatible with file installables due to argument ordering. For expressions, the feature doesn't make a lot of sense, because you can already put everything you need into the expression.

With other commands operating on installables

nix shell and nix build are the obvious cases we care about. nix profile install is a non-obvious case, but should work as well. nix copy should also work. nix edit might work, but I doubt it will ever be used. nix eval makes sense in some cases for debugging and should. nix run seems nonsensical, but maybe there's a good use-case?

Man that got long 😅 let me know what you think!

[viperML]

Thanks for the writeup @iFreilicht , LGTM . Personally I think --apply would be more meaningful than --expr, which has already an use. And as you said, perhaps having a different flag with the behaviour of the --expr examples, could be useful.

nix profile install is a non-obvious case, but should work as well

I would be concerned about the manifest.json, which follows a schema that might need to be updated. The schema is used to properly update the packages by referencing the same flakeref used to install it, so any --apply should be stored too. Personally I can live without nix profile --apply for the time being :)

$ nix profile install nixpkgs#python3 --profile ./test_profile

$ jq . ./test_profile/manifest.json
{
  "elements": [
    {
      "active": true,
      "attrPath": "legacyPackages.aarch64-linux.python3",
      "originalUrl": "flake:nixpkgs",
      "outputs": null,
      "priority": 5,
      "storePaths": [
        "/nix/store/idpjrasbr9n8kij11s5mphrw770sf13s-python3-3.10.12"
      ],
      "url": "path:/nix/store/xjviahzwa7x51vl51kc3c1k1n1jmhpd5-source?lastModified=1697059129&narHash=sha256-9NJcFF9CEYPvHJ5ckE8kvINvI84SZZ87PvqMbH6pro0%3D&rev=5e4c2ada4fcd54b99d56d7bd62f384511a7e2593"
    }
  ],
  "version": 2
}

[iFreilicht]

I would be concerned about the manifest.json, which follows a schema that might need to be updated.

The manifest format already supports store paths with barely any additional info. For example, I compiled nix locally and installed it with nix profile install ./result:

   {
      "active": true,
      "priority": 4,
      "storePaths": [
        "/nix/store/vfzyrg15gf2gvzi76ijsgjd004gcrncv-nix-2.19.0pre20231026_79e0c5c"
      ]
    }

So while an element like this can't be upgraded, it can absolutely be installed without any changes to the manifest format. And as all the installables produced by either --apply or --with would just be store paths, they can be treated as such in every context where installables are valid.

I don't want to advocate for "extensive support" for this kind of thing though, I'd much rather people write a flake with everything they want to have in their profile and install that.