source-repository-package doesn't work with nix 2.4

[TerrorJack]

Building a cabal project with source-repository-package in nix 2.4 will fail to fetch the repo with something like:

trace: No index state specified for haskell-project, using the latest index state that we know about (2021-11-08T00:00:00Z)!
trace: WARNING: No sha256 found for source-repository-package phadej/gentle-introduction.git 176cddab26a446bea644229c2e3ebf9e7b922559 download may fail in restricted mode (hydra)
fatal: couldn't find remote ref refs/heads/176cddab26a446bea644229c2e3ebf9e7b922559
error: program 'git' failed with exit code 128

[purefn]

I'm running into this with 2.6.1 too. Is there any known workaround?

[michaelpj]

I think there are some known bugs in 2.6.1 with restricted mode? Can't remember right now.

[TerrorJack]

I think it's related to some change of builtins.fetchGit since 2.4. Right now the workaround is to add sha256 annotation.

[purefn]

The problem we're running into with adding the sha256 is that we have several repos that are private. So adding the sha256 means that haskell.nix switches to using pkgs.fetchgit instead of builtins.fetchGit, which means it does not run at evaluation time and does not have access to the ssh directory of the user that is running the nix command.

Unfortunately this leaves us a bit stuck. The strange thing (to me) is that we have a stack.yaml file for the same project and using stackProject works and we run into this issue only for cabalProject. Do the two use builtins.fetchGit differently in some way?

[purefn]

To answer my own question:

For stack, we do this

https://github.com/input-output-hk/haskell.nix/blob/d420709339422e67fcdef4bf59d274cc620a88f7/lib/stack-cache-generator.nix#L98

                builtins.fetchGit ({
                  inherit (dep) url rev;
                } // pkgs.lib.optionalAttrs (branch != null) { ref = branch; });

For cabal, we do this

https://github.com/input-output-hk/haskell.nix/blob/d420709339422e67fcdef4bf59d274cc620a88f7/lib/call-cabal-project-to-nix.nix#L202

builtins.fetchGit { inherit (repoData) url ref; };

So for stack we always use rev, possibly adding a ref if there is a branch mapping for the commit. For cabal we always use ref, because we don't know if the tag for source-repository-package is a rev or a ref. But nix is expecting the ref attribute to be an actual ref in the repo, not a revision.

Something I didn't know is that you can specify a branch for source-repository-packages. The problem with it is that if tag and branch are defined, cabal-install always uses the latest HEAD of the branch, completely ignoring the tag value. What I'm wondering is if maybe we change what we do for cabal: if there is a tag, we assume it is a revision and pass it as rev to fetchGit. If there is a branch defined, we ignore the tag if there is one, similar to how cabal-install works, and pass the branch as ref.

The alternatives are to try and guess if the tag value looks like a commit based on some heuristics or allow some kind of branch lookup similar to what is done in stack.

[purefn]

FWIW it looks like the expectation in cabal is that tag is really intended to mean revision https://github.com/haskell/cabal/issues/6888. So maybe just assuming that the tag value can be used as rev and not ref would be reasonable.

[purefn]

I'm also wondering if we could eliminate the use of pkgs.fetchgit entirely. builtins.fetchGit supports a allRefs attribute, which, when true, will fetch all refs and not just the default. This would also eliminate the need for maintaining sha256 maps.

The biggest downside is that this could significantly increase the time and space to make these clones. But this is already the behavior of cabal and stack, so devs are used to it. They end up being surprised and frustrated that haskell.nix doesn't do it this way and they have to specify sha256 values to get things to work.

[michaelpj]

Does nix finally consider builtins.fetchGit with no sha to be pure/allowed in restricted mode? For a long time it didn't, and you needed to do pkgs.fetchgit because it's a fixed-output-derivation.

[purefn]

Ah no, it doesn't. I had forgotten about that detail.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hamishmack]

I am hoping that #1715 resolved this issue. Please reopen or open a new issue if I am mistaken.