haskell.nix
haskell.nix copied to clipboard
input-output-hk
Avoid the need for materialisation
I'm looking at https://github.com/input-output-hk/plutus/tree/master/nix/pkgs/haskell/materialized-linux to understand materialisation properly. If I have any misunderstandings please do say.
AFAICT, materialisation is simply a way to skip a bunch of IFD that haskell.nix does to figure out what to build. AFAICT, updating materialisation files is a PITA for newcomers to the ecosystem, not to mention that it's error-prone AFAICT since failing to update it will mean you are caching something wrong (albeit you could check this in CI I suppose).
I'm trying to understand the underlying motivations here and why it was done this way.
I'm thinking, why do we even need this? Couldn't all this materialisation stuff be a vanilla Nix derivation that outputs Nix code, and have that derivation depend only on the *.cabal files and similar?
I've been assigned to https://input-output.atlassian.net/browse/OBT-325 so I'd be doing all the work on this, though of course I'd force you to review my changes unfortunately.
@angerman @hamishmack @michaelpj (I assume you are the primary maintainers of haskell.nix?)
AFAICT, materialisation is simply a way to skip a bunch of IFD that haskell.nix does to figure out what to build.
Yes. Empirically, evaluation times with haskell.nix were a huge problem for people. It made using nix interactively very slow, and more importantly, it made Hydra slow to evaluate, meaning longer turnaround times on e.g. PR builds.
Materialization is therefore a big DevX improvement for most people most of the time; the cost is maintaining it. But we have that reasonably streamlined: in plutus you just have to run a single script and it fixes everything.
Couldn't all this materialisation stuff be a vanilla Nix derivation that outputs Nix code, and have that derivation depend only on the *.cabal files and similar?
That's what it is!
albeit you could check this in CI I suppose
And we do.
You might want to read the haskell.nix docs also: https://input-output-hk.github.io/haskell.nix/tutorials/materialization.html
Also I can't see that JIRA issue so I have no idea what you're referring to. Perhaps you could explain what you're trying to do and why?
So I'm wondering, why do we need to update the materialisation result manually?
Is it because the cost of evaluating the materisaliation derivation itself is expensive?
I don't understand the question. The output is checked into git, so you have to actually generate it and commit it. Generating it is automated.
What would happen if we rather than getting the materialisation file from the git repo, just generate it with a Nix derivation, and pass that to haskell.nix? That way we avoid committing the generated data into the repo. Is there something I'm missing wrt. why this wouldn't work?
That's called... IFD. That's what happens if you just don't use materialisation. But then it's slow.
OK, so there's no fundamental reason to it being slow, besides it just being unoptimised then I assume? This should theoretically be very simple. You just need to make sure that evaluating the expression for the derivation is simple. I'll put work into making this faster then.
I don't think we even need Recursive Nix for this.
You just need to make sure that evaluating the expression for the derivation is simple.
It calls cabal, it's never going to be fast.
You can make sure it's cached if the *.cabal files etc. haven't changed. And you need to update the materialisation if you do that anyway.
The potentially problematic issue here is that we need to filter out all the irrelevant files from the source, and unfortunately we might need ca-derivations to detect that nothing has changed. Are you OK with depending on this (optionally) to speed up builds? It would look something like cabalProject { useExperimentalNixFeatures = true; ... } probably.
IFD is never cached unless you make it fixed-output, I believe (which you can also do, but then you have to update the sha). So it will always rerun.
I think you're thinking precisely of the issue I just noted. The solution in current stable Nix is to fix the hash and make it content-addressed that way. The real solution is just to use ca-derivations, which removes the need to fix the hash.
We already filter the source, we're not crazy :) maybe you should look at the actual implementation first?
You filter the source at evaluation-time AFAICT, which is part of what might be making it slow potentially.
I'm not sure if I understand how CA derivations help. Could you make a small example showing that if you use CA derivations you can cache the result of something that depends on IFD? I don't see how that can work given that the IFD is potentially non-deterministic.
I'll try making a PoC that shows how you can do materialisation without it being a PITA then!
I'd be interested to just see something with a toy derivation that shows the principle. Doing a full PoC with materialization will be quite a bit of work.
Has this gone anywhere over the last month? Materialisation is really just a caching mechanism. Materialization for cabal files only is the easy part. And that's not really the major hit you get from the IFDs. The major hit you get from having to run cabal to produce build plans. For this you need cabal, and you need a ghc for the target.
I'd be happy for any improvement ins this regard.
We used to have various "regen.sh" scripts across repositories that effectively did materialisation for projects. The built-in materialisation support, baiscally means it's principled and similar across all repos. On top of that @hamishmack managed to impelemtn the checkMaterialized flag, to automatically verify that the materialization matches the expectation.
In an ideal world, the whole materialization logic shouldn't exit.
I took a look at the haskell.nix code base but I haven't made any tangible progress due to the complexity involved, albeit it should still be completely possible from what I can tell to remove the need entirely. I will probably have to read through the entire code base once to figure it out.
I'm not sure if I understand how CA derivations help.
I'll try to explain it in a bit more detail so maybe we can skip the POC.
The current logic probably is something like:
source -> fitlered-source -> expensive-IFD
Theoretically there is no need to re-compute the expensive-IFD as long as the output of filtered-source doesn't change.
But because nix is input addressed, the name+hash of filtered-source changes whenever source changes. So it changes all the time.
If content addressing was used, then the name+hash of filtered-source only changes whenever its actual output changes. Therefore the expensive-IFD wouldn't have to be computed all the time.
Content addressing basically removes the need of materialization.
I'd still like to see a tiny POC, could just be something with two derivations or something. I'm suspicious that Nix will re-do the IFD more often than that, but I don't have evidence of that. Easy enough to settle by just writing a tiny example.
So the whole discussion is abound filter source. Yes, we'd still need IFDs, but if we can fix that filter source in a content addressable way, we'd get more caching benefits.
Maybe it would be sensible to think about including only the IFD relevant files and the source on separate branches that are reconsidered during build.
Maybe it would be sensible to think about including only the IFD relevant files and the source on separate branches that are reconsidered during build.
Don't we do that already? I think we only include the cabal files when we're constructing the build plan, which is most of the IFD.
Don't we do that already? I think we only include the cabal files when we're constructing the build plan, which is most of the IFD.
Yes we do https://github.com/input-output-hk/haskell.nix/blob/eb49a3b7213470e36570f4aa1ed7a64e6d6cf160/lib/call-cabal-project-to-nix.nix#L111-L113
I think we will have to change that to always filter though (currently we only filter when haskellLib.canCleanSource is true).
In that case I'm even less convinced there are many gains to be found. But I'm looking forward to any improvements!
Interesting discussion.... I think the only gains haskell.nix is blowing it away entirely..... @DavHau whats the chances of dream2nix coming up with better ways to build cabal/stack/whatever haskell projects. Maybe some of the minds here can take a look what needs to be done.
I wonder if @iohk-devops would finance this work, directly or indirectly. I think the reliance on haskell.nix might be quite a bad thing long term, especially as the software seems to be getting more complex, with more dependencies , and more of everything... it will either break completely, or be so slow it will be useless.
However haskell.nix might be the best thing since sliced bread?
haskell.nix solves many problems. It's always tempting to say "we could write this again, but simpler!", but you often find you have to solve many of the same problems and it becomes complex again. Unless there's a clear argument for why a "new" version would be simpler, it doesn't seem that plausible to me.
I get
$ nix develop
trace: No index state specified for haskell-project, using the latest index state that we know about (2022-02-15T00:00:00Z)!
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
when updating only the non-cabal files. This doesn't need to happen AFAICT if the filtering worked correctly. I'm not sure where the exact issues is. Maybe the issues is that the flake is copied to a store path? Maybe it's not filterable then.
In any case, I firmly believe that we can fix haskell.nix such that materialisation can be removed entirely and still have the result will be strictly better for all users of haskell.nix.