Does niv actually prefetch downloads?

[michaelpj]

I'm a bit confused. I thought niv would use nix-prefetch-git or similar to get the hash that it puts into sources.json, which should also put the downloaded source in the store, so you shouldn't need to download it again later.

But I've repeatedly observed doing niv update foo; nix build ... and then observing foo being downloaded. Which seems odd.

[nmattia]

Yes, in most cases it uses nix-prefetch-url. I've also noticed this; I'm not quite sure what's happening, though I'm not sure either how nix-prefetch-url works. Would need to dig into this.

[duairc]

This is because the --name that you pass in the Haskell code to nix-prefetch-url is sanitizeName basename, while the name you set in sources.nix is sanitiizeName name + "-src". Remove the + "-src" and it will work.

[nmattia]

Oh wow, thank you!

[colinxs]

Just ran across this issue myself. Is the fix proposed by @duairc correct?

Thanks!

[exarkun]

It would be great to have this fixed. Normally the redundant download is merely annoying. Today GitHub is flaky and it takes 30+ minutes to download nixpkgs rendering most of my niv-based workflows completely broken.

[nmattia]

@exarkun can you confirm @duairc 's hypothesis above by applying the following diff to any of your sources.nix?

diff --git a/nix/sources.nix b/nix/sources.nix
index 1938409..7054127 100644
--- a/nix/sources.nix
+++ b/nix/sources.nix
@@ -8,7 +8,7 @@ let
 
   fetch_file = pkgs: name: spec:
     let
-      name' = sanitizeName name + "-src";
+      name' = sanitizeName name;
     in
       if spec.builtin or true then
         builtins_fetchurl { inherit (spec) url sha256; name = name'; }
@@ -17,7 +17,7 @@ let
 
   fetch_tarball = pkgs: name: spec:
     let
-      name' = sanitizeName name + "-src";
+      name' = sanitizeName name;
     in
       if spec.builtin or true then
         builtins_fetchTarball { name = name'; inherit (spec) url sha256; }

[exarkun]

I did this:

$ niv init
Initializing
  Creating nix/sources.nix
  Creating nix/sources.json
  Importing 'niv' ...
  Adding package niv
    Writing new sources file
  Done: Adding package niv
  Importing 'nixpkgs' ...
  Adding package nixpkgs
    Writing new sources file
  Done: Adding package nixpkgs
Done: Initializing

and after 20 or 30 minutes ended up with this sources.json:

{
    "niv": {
        "branch": "master",
        "description": "Easy dependency management for Nix projects",
        "homepage": "https://github.com/nmattia/niv",
        "owner": "nmattia",
        "repo": "niv",
        "rev": "9cb7ef336bb71fd1ca84fc7f2dff15ef4b033f2a",
        "sha256": "1ajyqr8zka1zlb25jx1v4xys3zqmdy3prbm1vxlid6ah27a8qnzh",
        "type": "tarball",
        "url": "https://github.com/nmattia/niv/archive/9cb7ef336bb71fd1ca84fc7f2dff15ef4b033f2a.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },
    "nixpkgs": {
        "branch": "release-20.03",
        "description": "Nix Packages collection",
        "homepage": "",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "eb73405ecceb1dc505b7cbbd234f8f94165e2696",
        "sha256": "06k21wbyhhvq2f1xczszh3c2934p0m02by3l2ixvd6nkwrqklax7",
        "type": "tarball",
        "url": "https://github.com/NixOS/nixpkgs/archive/eb73405ecceb1dc505b7cbbd234f8f94165e2696.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    }
}

Then I applied the above patch to sources.nix and in nix repl evaluated:

(import ./nix/sources.nix).nixpkgs

but alas it immediately began downloading:

[1.7 MiB DL] downloading 'https://github.com/NixOS/nixpkgs/archive/eb73405ecceb1dc505b7cbbd234f8f94165e2696.tar.gz'

[tfc]

Also tried it right now. The proposed fix does not help.

[infinisil]

[pid 97301] execve("/run/current-system/sw/bin/nix-prefetch-url", ["nix-prefetch-url", "--unpack", "https://github.com/infinisil/nix"..., "--name", "3d829c74b3dff74f211ef21d64def233"...], 0x7ffde1e29e38 /* 147 vars */ <unfinished ...>

nix-prefetch-url is called with the basename of the url, not the name of the input!