shellify use the local nixpkgs path instead of the remote one

[RoadRoller01]

i am using flake for my nixos, and set the nixpkgs to "github:nixos/nixpkgs/nixos-unstable" so it should use it instead of the local one.

[danielrolls]

Hi @RoadRoller01, the value for that input is actually taken from the output of nix flakes registry list. If you run nix shell nixpkgs#someProgram you'll get someProgram from whatever nixpkgs is set to in the registry. nix-shell -p someProgam will instead take someProgram from the wherever nixpkgs points but AFAIK flakes can't point at this environment variable. I have to admit I didn't know all this until I wrote this tool.

I could add functionality to try to in some circumstances decode the value of nixpkgs and attempt to convert it to a flake input. But I wonder if it's overkill, especially since using nix-shell is the old way now. I tried to get a feeling of what the community want on discourse but didn't get a lot of discussion. Please consider contributing there if you have ideas as it's not clear to me what the consensus is on this from the community so I'm keen to see discussion on the matter: https://discourse.nixos.org/t/shellify-tool-for-shell-nix-generation/33935

Also, it would be interesting to know why your nixpkgs registry entry is a local path. My nixos is built from flakes too and it doesn't look like that. Do you happen to know why yours is like that? Can you please share your nix registry list output?

Thanks for the feedback.

[RoadRoller01]

sure, its

system flake:nixpkgs path:/nix/store/801l7gvdz7yaibhjsxqx82sc7zkakjbq-source
global flake:agda github:agda/agda
global flake:arion github:hercules-ci/arion
global flake:blender-bin github:edolstra/nix-warez?dir=blender
global flake:bundlers github:NixOS/bundlers
global flake:cachix github:cachix/cachix
global flake:composable github:ComposableFi/composable
global flake:disko github:nix-community/disko
global flake:dreampkgs github:nix-community/dreampkgs
global flake:dwarffs github:edolstra/dwarffs
global flake:emacs-overlay github:nix-community/emacs-overlay
global flake:fenix github:nix-community/fenix
global flake:flake-parts github:hercules-ci/flake-parts
global flake:flake-utils github:numtide/flake-utils
global flake:gemini github:nix-community/flake-gemini
global flake:helix github:helix-editor/helix
global flake:hercules-ci-agent github:hercules-ci/hercules-ci-agent
global flake:hercules-ci-effects github:hercules-ci/hercules-ci-effects
global flake:home-manager github:nix-community/home-manager
global flake:hydra github:NixOS/hydra
global flake:mach-nix github:DavHau/mach-nix
global flake:nickel github:tweag/nickel
global flake:nimble github:nix-community/flake-nimble
global flake:nix github:NixOS/nix
global flake:nix-darwin github:LnL7/nix-darwin
global flake:nix-serve github:edolstra/nix-serve
global flake:nixops github:NixOS/nixops
global flake:nixos-hardware github:NixOS/nixos-hardware
global flake:nixos-homepage github:NixOS/nixos-homepage
global flake:nixos-search github:NixOS/nixos-search
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
global flake:nur github:nix-community/NUR
global flake:patchelf github:NixOS/patchelf
global flake:poetry2nix github:nix-community/poetry2nix
global flake:pridefetch github:SpyHoodle/pridefetch
global flake:sops-nix github:Mic92/sops-nix
global flake:systems github:nix-systems/default
global flake:templates github:NixOS/templates

for some reason i have two nixpkgs one tagged global and one tagged system, so i think you should use the global one?

[danielrolls]

Thanks for sharing the output.

If I were to use the system one this would break for somebody choosing to override the global one with their own local version. It would be good to understand how the local one got there. You might want to try updating your Nixos flake and see if the system nixpkgs changes. If it doesn't then you should delete it because it is surely growing stale. If it changes I suspect you are using something that chooses to pin nixpkgs to a local version and my gut feeling is that shellify should respect that rather then do something confusing or convoluted.

I'm interested to know if your flake did this and whether others are likely to get have situation.

[RoadRoller01]

i quit nixos(:

[Vaisakhkm2625]

i quit nixos(:

:( nooooo..... i was about to ask if you could add a flag to use global values...

[danielrolls]

@Vaisakhkm2625, I'm now seeing the system value set to a Nix store path too. I think this behavior may have changed. I now suspect we should ignore the system value for at least nixpkgs by default and not with a flag. It would be good to understand this better though. Any experiments or analysis in this area are most welcome.

[Vaisakhkm2625]

@Vaisakhkm2625, I'm now seeing the system value set to a Nix store path too. I think this behavior may have changed. I now suspect we should ignore the system value for at least nixpkgs by default and not with a flag. It would be good to understand this better though. Any experiments or analysis in this area are most welcome.

so can we reopen the issue to make it global?

[danielrolls]

@Vaisakhkm2625, I'm now seeing the system value set to a Nix store path too. I think this behavior may have changed. I now suspect we should ignore the system value for at least nixpkgs by default and not with a flag. It would be good to understand this better though. Any experiments or analysis in this area are most welcome.

so can we reopen the issue to make it global?

Yes, of course we can. The main thing I think we need to understand to address this is whether the pinning of the system value is happening for nixpkgs only or whether we need to change the rules for all inputs. I don't want to make the situation amny more complicated than it already is.

[danielrolls]

I'm part way through reading the comments in https://github.com/NixOS/nix/issues/7422?utm_source=pocket_saves which may help us understand the situation better.

[danielrolls]

So I read the above and found a link to https://github.com/NixOS/nixpkgs/pull/254405/files which appears to be pinning the system flake specifically for the case of nixpkgs. I don't think it makes sense to copy this pinning over into flake.nix (although pinning flake.lock would be beneficial). I'm thinking we should do the following for now:

1. By default take from the user registry, then system if not available and finally from global.

2. In the case of nixpkgs only, if the system entry is set to point to a path then fall straight through to the global entry.

3. Keep the above by default but add a switch --always-take-from-system-registry to revert to the old behavior.

@Vaisakhkm2625 what do you think? This is more complicated than I would like but I think it's necessary due to the complicated implementation of nix registries in the nix tool.

[Vaisakhkm2625]

So I read the above and found a link to https://github.com/NixOS/nixpkgs/pull/254405/files which appears to be pinning the system flake specifically for the case of nixpkgs. I don't think it makes sense to copy this pinning over into flake.nix (although pinning flake.lock would be beneficial). I'm thinking we should do the following for now:

1. By default take from the user registry, then system if not available and finally from global.

2. In the case of nixpkgs only, if the system entry is set to point to a path then fall straight through to the global entry.

3. Keep the above by default but add a switch `--always-take-from-system-registry` to revert to the old behavior.

@Vaisakhkm2625 what do you think? This is more complicated than I would like but I think it's necessary due to the complicated implementation of nix registries in the nix tool.

Yes, exactly what i would like to see... if user is using pkgs different from the global, it doesn't make sense to directly pull from the global, so using user registry by default would be perfect.. but without the pinning

[danielrolls]

Great. Thanks for the feedback. It might come after Christmas I'm afraid but it is high on my todo list.

[danielrolls]

@Vaisakhkm2625, just so you know this is happening, I've started here: https://github.com/danielrolls/shellify/pull/17

It already deprioritises local pinned repositories so feel free to try it out. I need to add the --always-take-from-system-registry flag and clean up a little before I merge.

[Vaisakhkm2625]

wow, thanks for that.. but i am getting some error, prob i shouldn't have took the latest one

[Vaisakhkm2625]

both

nix run "github:danielrolls/shellify/avoid-local-pinned-registry-urls"

 nix run github:danielrolls/shellify/b35ce7ad3191a8cdf2f13933c0cf62a80f6fbc3f

not working for me... any idea?

[danielrolls]

both

nix run "github:danielrolls/shellify/avoid-local-pinned-registry-urls"

 nix run github:danielrolls/shellify/b35ce7ad3191a8cdf2f13933c0cf62a80f6fbc3f

not working for me... any idea?

Appologies. I should have said. I'm building on the avoid-local-pinned-registry-urls branch using:

nix develop --command cabal v2-update
nix develop --command cabal v2-build

Once I'm done I'll check it against nixpkgs and loosen any bounds that need loosening to distribute. Thanks for verifying it. I'll aim to finish it next weekend. No promises though!

[Vaisakhkm2625]

Sorry for the delay in the response...

i have no clue about haskell build system, but after running cabal v2-build, i found executable in /dist-newstyle/build/x86_64-linux/ghc-9.2.8/shellify-0.11.0.5/x/nix-shellify/build/nix-shellify/nix-shellify

so i tired with shellify/dist-newstyle/build/x86_64-linux/ghc-9.2.8/shellify-0.11.0.5/x/nix-shellify/build/nix-shellify/nix-shellify shell nixpkgs\#jdk17 nixpkgs\#jetbrains.idea-community and worked...

But currently it's using nixpkgs-unstable branch even though i hope i specified stable one... is this intentional?

{
  description = "my project description";

  inputs = {

    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let nixpkgsPkgs = if builtins.hasAttr "packages" nixpkgs then nixpkgs.packages.${system} else ( if builtins.hasAttr "legacyPackages" nixpkgs then nixpkgs.legacyPackages.${system} else nixpkgs);
        in
        {
          devShells.default = import ./shell.nix { pkgs=nixpkgsPkgs; };
        }
      );
}

Steps i tried

git clone [email protected]:danielrolls/shellify.git
git checkout avoid-local-pinned-registry-urls

nix develop --command cabal v2-update
nix develop --command cabal v2-build

find dist-newstyle -type f -executable

cd ..

shellify/dist-newstyle/build/x86_64-linux/ghc-9.2.8/shellify-0.11.0.5/x/nix-shellify/build/nix-shellify/nix-shellify shell nixpkgs\#jdk17 nixpkgs\#jetbrains.idea-community

also let me know if you needed any help

I'll aim to finish it next weekend. No promises though!

:) i will wait.. thanks for taking time to fix this..

[danielrolls]

@Vaisakhkm2625,

If it's picking up nixpkgs-unstable it's from the output of nix registry list. If you run nix registry list | grep nixpkgs you can see what's set for you. For me it's set to to use nixpkgs-unstable so I suspect we all have that by default.

I added that flag as planned. I changed the name. There's bits I want to tidy but if it all works I'm tempted to release it to get the fix out. Let me know what you think.

BTW, after I round off this work I'm tempted to explore adding support for using niv instead of flakes. I haven't used niv yet but I can see that flakes are more complicated than I'd have hoped!

[Vaisakhkm2625]

Ho, i never heard of niv... nix has already issues with setting up things like various c libs, esp for python enviroments (like qt,numpy nightmares).. and needing override some random env variables.. is already nighmare... which alway get more complicated when i use some tools to make it easy, like devenv and pypoetry... so i give up on such tools...

but i will try this niv for some time anyway... thanks for suggetion

[danielrolls]

Ok. So I released the fix onto Hackage: https://hackage.haskell.org/package/shellify

nixpkgs will pick it up and bring it automatically into nixpkgs-unstable. You're better off always taking shellify from unstable. The process is very slow though and run by a small number of volunteers.

[Vaisakhkm2625]

Thanks.. :) anyway i am using the locally compiled binary for now..

[danielrolls]

@Vaisakhkm2625, FYI. nix run github:danielrolls/shellify -- --help works now!

Are you happy for me to close this issue?

[Vaisakhkm2625]

thanks.. it's working.. :) really thanks for the hard work you done..