Push bulk of sources.nix generated via niv init into a canonical URL

[masaeedu]

Hi there. After running niv init today, I get the following code in my nix/sources.nix:

# This file has been generated by Niv.

# A record, from name to path, of the third-party packages
with rec
{
  pkgs =
    if hasNixpkgsPath
    then
        if hasThisAsNixpkgsPath
        then import (builtins_fetchTarball { inherit (sources_nixpkgs) url sha256; }) {}
        else import <nixpkgs> {}
    else
        import (builtins_fetchTarball { inherit (sources_nixpkgs) url sha256; }) {};

  sources_nixpkgs =
    if builtins.hasAttr "nixpkgs" sources
    then sources.nixpkgs
    else abort
    ''
        Please specify either <nixpkgs> (through -I or NIX_PATH=nixpkgs=...) or
        add a package called "nixpkgs" to your sources.json.
    '';

  # fetchTarball version that is compatible between all the versions of Nix
  builtins_fetchTarball =
      { url, sha256 }@attrs:
      let
        inherit (builtins) lessThan nixVersion fetchTarball;
      in
        if lessThan nixVersion "1.12" then
          fetchTarball { inherit url; }
        else
          fetchTarball attrs;

  # fetchurl version that is compatible between all the versions of Nix
  builtins_fetchurl =
      { url, sha256 }@attrs:
      let
        inherit (builtins) lessThan nixVersion fetchurl;
      in
        if lessThan nixVersion "1.12" then
          fetchurl { inherit url; }
        else
          fetchurl attrs;

  # A wrapper around pkgs.fetchzip that has inspectable arguments,
  # annoyingly this means we have to specify them
  fetchzip = { url, sha256 }@attrs: pkgs.fetchzip attrs;

  hasNixpkgsPath = (builtins.tryEval <nixpkgs>).success;
  hasThisAsNixpkgsPath =
    (builtins.tryEval <nixpkgs>).success && <nixpkgs> == ./.;

  sources = builtins.fromJSON (builtins.readFile ./sources.json);

  mapAttrs = builtins.mapAttrs or
    (f: set: with builtins;
      listToAttrs (map (attr: { name = attr; value = f attr set.${attr}; }) (attrNames set)));

  # borrowed from nixpkgs
  functionArgs = f: f.__functionArgs or (builtins.functionArgs f);
  callFunctionWith = autoArgs: f: args:
    let auto = builtins.intersectAttrs (functionArgs f) autoArgs;
    in f (auto // args);

  getFetcher = spec:
    let fetcherName =
      if builtins.hasAttr "type" spec
      then builtins.getAttr "type" spec
      else "builtin-tarball";
    in builtins.getAttr fetcherName {
      "tarball" = fetchzip;
      "builtin-tarball" = builtins_fetchTarball;
      "file" = pkgs.fetchurl;
      "builtin-url" = builtins_fetchurl;
    };
};
# NOTE: spec must _not_ have an "outPath" attribute
mapAttrs (_: spec:
  if builtins.hasAttr "outPath" spec
  then abort
    "The values in sources.json should not have an 'outPath' attribute"
  else
    if builtins.hasAttr "url" spec && builtins.hasAttr "sha256" spec
    then
      spec //
      { outPath = callFunctionWith spec (getFetcher spec) { }; }
    else spec
  ) sources

Is it possible to move the bulk of this into some canonical location in the niv repo, and instead have niv init generate something like:

import (fetchUrl https://github.com/nmattia/niv/<some commit hash>/whatever.nix)

? Or better yet, just have us use that directly in our default.nix, and not generate anything besides the sources.json in our version controlled code.

[nmattia]

Hi there. After running niv init today, I get the following code in my nix/sources.json:

You mean nix/sources.nix, correct?

Is it possible to move the bulk of this into some canonical location in the niv repo, [...]

Oh, this is a lovely idea. The import as is is not going to work, but I could imagine something like this:

let
  sourcesFun = import (builtins.fetchurl
    { url = https://raw.githubusercontent.com/nmattia/niv/1dd094156b249586b66c16200ecfd365c7428dc0/nix/sources_fun.nix; });
  sources = sourcesFun ./nix/sources.json;
  pkgs = import sources.nixpkgs {};
in pkgs.runCommand "foo" {}
  "echo hello > $out"

Nix purists are going to hate it, but... I like it! Would could even supply a sha256 to fetchurl to make everyone happy.

@nlewo how's that going to play with Hydra? builtins.fetchurl is going to fail, correct? We could probably lift the <nixpkgs> hack out of sources.nix and do something like let bootstrapPkgs = import <some-nixpkgs> {}; in pkgs.fetchurl ...

@basvandijk @zimbatm @timbertson thoughts on this?

[nmattia]

Mh, actually this would also trivially solve https://github.com/nmattia/niv/issues/102, since we'd be passing the path to sources.json explicitly. @masaeedu you seem to have an agenda here... :wink:

[masaeedu]

[masaeedu]

whoops, fatfingered the close button.

You mean nix/sources.nix, correct?

Yeah, my bad.

[leotaku]

Mh, actually this would also trivially solve #102, since we'd be passing the path to sources.json explicitly. @masaeedu you seem to have an agenda here... wink

So essentially the idea is to scrap the nix folder entirely and have all the action inside the default.nix file from the online repository? If that is the case, how would "updating" an outdated sources.nix file work? Currently we get a notification from niv when there is a new version, but this change would make such a thing impossible, right?

In any case, I think this is actually a pretty good idea, but I would like to ask for a command/flag that just automatically downloads the correct sources.nix file for my version of niv. I prefer just having said file in my VC, instead of having to download and import it from the web. (I'm also curious, why do you guys prefer the latter approach?) I think it would also solve the hydra problem.

[nmattia]

If that is the case, how would "updating" an outdated sources.nix file work?

The first thing I can come up with is keeping a list of know shas of the sources.nix in niv and updating that in the default.nix (alongside the git rev). There's a problem however since we give more freedom to the user and they may store the hash/fetchurl anywhere.

I prefer just having said file in my VC

I'm thinking of

$ niv init --remote-sources-nix

or similar during bootstrap, which would enable the new behavior. The old one should still be supported.

why do you guys prefer the latter approach?

Let stuff to check in! Though it'll be harder for people to just tweak the sources.nix.

[masaeedu]

@leotaku Wouldn't it be easier to update an outdated sources.nix file with this approach? I (and everyone else who uses niv) just need to change the commit SHA to get a future version. The maintenance of the contents of sources.nix is centralized, so there's not a hundred different versions copy pasted into everyone's repos across github.

[leotaku]

@masaeedu As @nmattia has already mentioned, his proposed approach to import everything into files that are edited by the user (e.g. default.nix) would make updating automatically very janky.

As for simply replacing the existing sources.nix with a minimal version that imports code from the GitHub repo, I really don't understand the appeal. I personally only see disadvantages:

• Internet access required for building anything (IIRC local fetchers are a thing)
• Hydra issues
• Harder to debug/tweak sources.nix file
• Less understandable git diffs after updating
• Yet another "dependency"

[masaeedu]

@leotaku If you debug/tweak your sources.nix file, how do you usually update it? Wouldn't updating destroy your changes?

As far as my usage is concerned I don't usually tweak the sources.nix file myself; if I needed to I'd fork niv and use that as a submodule in my project. That way at least I'd be able to try and rebase my changes on top of future changes in the niv repo. This would also be the solution to avoiding internet access; I don't have to get the niv repo from the internet, I can get it from any nix derivation, including one that originates from my filesystem.

I actually already require internet access today for nix-shell-ing into my niv-based Haskell projects, I wasn't aware it was supposed to work without internet access. Maybe I have something misconfigured?

[masaeedu]

Less understandable git diffs after updating

I guess this is the fundamental difference in opinion. I actually don't want to see any niv related changes to source files when I'm browsing the history of my projects. I'd prefer to see niv updates as just a change in a SHA.

[leotaku]

If you debug/tweak your sources.nix file, how do you usually update it? Wouldn't updating destroy your changes?

@masaeedu Generally my changes to the sources.nix file are only experimental, so I don't really have this problem very often. But what I do is move the old sources.nix file, niv init, diff and manually resolve the problem.

I actually already require internet access today for nix-shell-ing into my niv-based Haskell projects, I wasn't aware it was supposed to work without internet access. Maybe I have something misconfigured?

I'm not quite sure, but from my limited testing it should work. I think it also depends on your fetchers, if they need to download something, you of course need internet access. Do you need internet access every time you nix-shell or just sometimes?

I guess this is the fundamental difference in opinion.

I couldn't agree more. I guess it comes from of me wanting as few moving parts as possible and only using niv in projects that already deal with a lot of Nix code. Most people probably just don't really need or want to think too much about niv. :)

What do you think about @nmattia's proposal to make this configurable via a flag. This way both of our ideal workflows could be supported.

[masaeedu]

Do you need internet access every time you nix-shell or just sometimes.

It's not something I encounter regularly, I was just on a flight last week without internet access and discovered I couldn't nix-shell into one of my niv-based repos for some reason. I just disconnected from internet and tried again and got:

➜  repos cd biparser
direnv: loading .envrc
direnv: using nix
warning: unable to download 'https://cache.dhall-lang.org/6vv6skb2gw1g2pifvaxr7ii1h9jsc23z.narinfo': Couldn't resolve host name (6); retrying in 313 ms

so maybe it's not niv related?

What do you think about @nmattia's proposal to make this configurable via a flag. This way both of our ideal workflows could be supported.

Yeah, I definitely agree with having a flag to make it configurable, I didn't mean to imply it should be impossible to recover the old behavior.

[timbertson-zd]

For reference, this is what nix-wrangle does: the generated default.nix just does some bootstrapping to import nix-wrangle from the version specified in your actual sources JSON. The nix code it imports (nix/wrangle/nix/api.nix) has the full logic around actually loading your JSON files and resolving each source.

It's still not a tiny bootstrap (~20 lines), but there shouldn't be much need to change it, so it won't give much diff noise after the initial commit. Although in niv's case there's less nix code to offload into a theoretical api.nix, the bootstrap code already does most of what's needed.

[nmattia]

Made some progress on this with @basvandijk. This sources.nix can be imported with builtins.fetchurl: https://raw.githubusercontent.com/nmattia/niv/506b896788d9705899592a303de95d8819504c55/nix/sources.nix 🎉

[nmattia]

Hey guys, would love your feedback on https://github.com/nmattia/niv/issues/272