haskell.nix No clean way of adding non-Hackage and non-Stackage Haskell dependencies

See #1339.

That PR is an attempt to enable this functionality, however, it's a hack, just like the package: ${pkg} approach @angerman proposes. Like @michaelpj wrote, we want a clean way of being able to include external dependencies, perhaps something like the following:

{
  pkg-def-extras = hackage: {
     myPackage = callCabalToNix ./myPackage.cabal;
  };
}

This is also relevant for #1262.

Jan 18 '22 09:01 L-as

Can you explain why source-repository-package doesn't do what you want?

Jan 18 '22 09:01 michaelpj

source-repository-package doesn't allow anything not fetchable by Cabal, i.e.

Non-git
Private repository
Source generated by Nix

In addition, it's a pain, and not easy to manage when you essentially need to have your own package index. E.g. in the Plutus ecosystem we do the whole copying cabal.project thing because we need to make sure we use the same version of e.g. cardano-prelude as upstream. This is all a huge PITA.

Jan 18 '22 13:01 L-as

These are all problems with Cabal, rather than haskell.nix IMO.

E.g. in the Plutus ecosystem we do the whole copying cabal.project thing because we need to make sure we use the same version of e.g. cardano-prelude as upstream. This is all a huge PITA.

The correct solution is for us to stop using a massive web of source-repository-packages and instead use a proper package repository, either Hackage or an internal one. Again, not a haskell.nix problem IMO.

Jan 18 '22 14:01 michaelpj

an internal one

😬 This would become one... but we should know in a few days, thanks to https://github.com/input-output-hk/hackage-overlay-ghcjs

Jan 18 '22 15:01 angerman

I don't see how such a package repository would work without something like hackage.nix specialised to it.

Jan 18 '22 16:01 L-as

@L-as i share similar concerns. The storage backing is different though. I'll see over the next few days.

Jan 18 '22 16:01 angerman

I am a Nix zealot, so I am quite biased, but I imagine you could define the repository in pure Nix, such that it works (only) with exactDeps = true;.

Jan 18 '22 16:01 L-as

@michaelpj This is another issue with source-repository-package: exactDeps doesn't work with it unless you don't have it in the shell and only while building the shell.

Jan 18 '22 16:01 L-as

I am a Nix zealot, so I am quite biased, but I imagine you could define the repository in pure Nix

IMO haskell.nix is a project for Nix-ifying cabal projects, not a fancy Nix-based Haskell builder that goes beyond cabal.

Jan 18 '22 16:01 michaelpj

I don't think that's just your opinion, that's the state of haskell.nix. I do concede that what I ultimately want is not haskell.nix, but something much more lightweight that rather than emulating what Cabal does and make it reproducible, presents a different Nix-native workflow, like I told you in London, however, I still believe that exposing the ability to use a custom Hackage essentially would not be very hard, and now that I think about it, it's probably the approach I should be going for.

Jan 18 '22 17:01 L-as

Allowing custom hackage repos is certainly something I wish we already had. It's also the way cabal handles this.

Why do we focus so much on cabal? Because that's our ground truth. We don't (and can't: Windows, BSD, ...) force everyone to use nix, this the repos should work fine without.

But we do want nix in CI and for cross compilation, and ...

So if it can be expressed in cabal, Haskell.nix should just take it.

It's the same reason why cabal.project.local is my preferred interface when possible. It's the same strategy the cabal uses. And it ensures we get the same behavior.

One thing where Haskell.nix deviates from cabal is that we don't support multiple plans well. Hence the nonReinstallabaldPkgs madness :-(

Jan 18 '22 23:01 angerman

Custom hackages already work, you just have to make them yourself: https://github.com/input-output-hk/haskell.nix/pull/535

Jan 19 '22 09:01 michaelpj

Ohh I completely forgot about that PR! 🤦‍♂️

Jan 19 '22 10:01 angerman

Here is a prototype for using extra-hackages and extra-hackage-tarballs to add dependencies by path. https://github.com/mlabs-haskell/haskell-nix-extra-hackage/tree/0a1061b6a9da11da045c543f13ce32af9897ac74 Possible additions to haskell.nix worth considering:

automatically add extra-hackages if extra-hackage-tarballs is provided but extra-hackages is not
new extraHackagePackages argument to *project that does it all

May 09 '22 21:05 brainrake

@angerman @michaelpj Forgot to note this, but do note that pkg-def-extras can not be used for this, as it doesn't effect plan generation, hence the IFD will fail and complain that the packages are unknown, and no plan will be generated.

May 20 '22 11:05 L-as

We had some issues with the above too, due to implicit ordering of hackages based on their names, which lead to Hackage taking precedence over the custom Hackage, meaning the compact-map IOG package couldn't be used, until we switched the names internally.

May 20 '22 11:05 L-as

We have a working prototype of using a custom Hackage inside IOG. It's just a matter of getting buy-in and doing it now.

May 20 '22 12:05 michaelpj

@michaelpj That would really be great. Do you plan to do something like the hackage.nix repo or something else?

May 21 '22 08:05 L-as

Also, I think we can remove pkg-def-extras, since it has no use AFAICT?

May 21 '22 08:05 L-as

Do you plan to do something like the hackage.nix repo or something else?

Better than that: the cabal repo itself is in a git repository, so you just pull it in as a flake input and do hackage-to-nix to it on the fly, so it's even simpler.

May 23 '22 09:05 michaelpj

For the sake of completeness: I solved a similar problem here: https://github.com/ilyakooo0/haskell-nix-extra-hackage

Jun 13 '22 06:06 ilyakooo0

version with separate hackage per package, to reduce rebuilds https://github.com/mlabs-haskell/haskell-nix-extra-hackage/tree/separate-hackages

Jun 13 '22 15:06 brainrake