Unable to use a `mirror:` URL for tarballs

[purcell]

Steps to reproduce

• niv add emacs -v 24.3 -t 'mirror://gnu/emacs/emacs-<version>.tar.gz'

Expected behaviour

• sources.emacs behaves the same as a regular tarball source

Actual behaviour

Evaluation fails:

while evaluating 'fetch' at /Users/steve/Projects/nix-emacs-ci/nix/sources.nix:70:23, called from /Users/steve/Projects/nix-emacs-ci/nix/sources.nix:118:31:
while evaluating 'fetch_file' at /Users/steve/Projects/nix-emacs-ci/nix/sources.nix:9:22, called from /Users/steve/Projects/nix-emacs-ci/nix/sources.nix:74:38:
while evaluating 'builtins_fetchurl' at /Users/steve/Projects/nix-emacs-ci/nix/sources.nix:101:23, called from /Users/steve/Projects/nix-emacs-ci/nix/sources.nix:11:7:
unable to download 'mirror://gnu/emacs/emacs-24.3.tar.gz': Unsupported protocol (1)

Discussion

This occurred in the course of trying to switch from use of builtins.fetchurl to niv: it seems that fetchurl handles mirror:// URLs, but fetchTarball does not. I wonder if Niv could use the former? I note that in nixpkgs, functions like fetchPypi use fetchurl to retrieve tarballs, in order to be able to use mirror:// URLs.

(Thanks for Niv: it's such a nice piece of software!)

[nmattia]

Yep, we could definitely use the builtin fetcher instead. Right now we default to using the pkgs. fetchers for reasons that I believe are legit but can't remember just now. What we could do is to parse the URI and basically turn on builtin = true (the flag that instructs niv to use the builtin fetcher) when the protocol is mirror://. WDYT?

[purcell]

Hmm, I tried adding "builtin": true to the source in sources.json, but that isn't sufficient, because the code still uses fetchTarball instead of fetchurl.

[purcell]

Fwiw, locally I patched sources.nix so that fetch_tarball delegates unconditionally to pkgs.fetchzip (which understands mirrors) instead of builtins_fetchTarball and this works well afaict. I don't know the background and original rationale of the builtin flag, but I wonder if it's even still necessary.

[nmattia]

I don't know the background and original rationale of the builtin flag, but I wonder if it's even still necessary.

Using the built-ins is better (and really the only solution) if you don't have nixpkgs; this means that out of the box, without downloading GBs of executables, you can unpack most dependencies.

In the case of mirrors, I guess that just doesn't work! There's another case where built-ins are unpractical: on Hydra, each URI (and hence URL) access needs to be explicitly allowed. Since pkgs.fetchX doesn't use builtins.fetchX but instead uses a (safer) fixed output derivation, URIs don't need to be allowed.

Hope this helps clarify the rationale!

[purcell]

Yeah, the rationale makes sense. I found that the actual expressions are something like

      if spec.builtin or true then
        builtins_fetchTarball { name = name'; inherit (spec) url sha256; }
      else
        pkgs.fetchzip { name = name'; inherit (spec) url sha256; };

but unless I'm wrong, this means the pkgs.fetchzip can never be used: there's no way that spec.builtin or true will ever be anything but true, right?

[nmattia]

It can!

nix-repl> spec = { }

nix-repl> spec.builtin or true
true

nix-repl> spec = { builtin = false; }

nix-repl> spec.builtin or true
false

nix-repl> spec = { builtin = true; }

nix-repl> spec.builtin or true
true

The semantics of or are that either the attribute is used (if it exists), or the RHS is used. The LHS attribute value is never read or used in the process, just returned (if I remember correctly).

[purcell]

Whoah, that's an unexpected behaviour! Thanks for taking the time to clarify.