cabal new-install, environment files, and unsolvable constraints

[typedrat]

Because new-install wants to keep every package that GHC bundles in the environment, it adds every package from this list to the environment along with the initial targets. Unfortunately, this can create environments that, when read back in, are not actually valid solved package sets, because (e.g.) Cabal is happy to pick a newer version of directory that is incompatible with the current ghc's dependency, and then create an environment that references both this newer directory version and the ghc library version that it cannot be used with, and further calls to new-install will fail because of this.

In addition, it does not correctly remove constraints on older versions of the target library, so if (e.g.) semigroups-0.18.4 has been new-installed and one attempts to install semigroups-0.18.5, this will also conflict because there is a hard constraint on the previous semigroups version.

[hvr]

this needs to have a solution/algorithm/heuristic designed to proceed

[typedrat]

Consider a sketch of a solution:

1. start with a notional package that depends on all the default libraries based on compiler version (can just look at global package env? maybe)
2. each new-install --lib is isomorphic to adding a new dependency to this package and then updating what we track, with same errors as normal if you try to add an unsolvable dependency to a package.

[23Skidoo]

For 3.0, maybe we should add a warning to install --lib that it's somewhat experimental for now?

[tomjaguarpaw]

I've just been bitten by this. cabal v2-install claimed that directory-1.3.4.0 was a user goal (which is false) and that directory-1.3.4.0 conflicts with the one that came with my GHC (which is true). Deleting the entry for directory-1.3.4.0(and several others) in my .ghc/<version>/environments/default allowed me to v2-install successfully. On the other hand the offending lines were recreated by that command!

Could someone please explain to me how those lines got there? Is cabal putting them there or is ghc putting them there as a consequence of how cabal calls it?

[TomMD]

This is perhaps the most common Haskell gotcha I walk though with my new devs and people from afar (such as here https://stackoverflow.com/questions/58755666/haskell-install-puremd5-package/58759535#58759535).

I'm not sure what the options are for a smooth universal solution... some brainstorming:

• Status quo (ugh)
• A new environment per project. Pretty sure this was a default for a moment in time but people didn't like ghci giving a different environment without warning and based on the CWD.
• A nicer error message that says the conflict is due to an environment and how to either remove the default (and that doing so is safe) or use a new environment.
• @typedrat's proposal where cabal try's to solve using the current environment as constraints.

I'm in favor of some combination of the last two. Are there other apparent options out there to consider?

[gbaz]

At least 3 should occur. I think some of the cabal-env stuff (https://github.com/phadej/cabal-env) is sort of supposed to prototype some workflows for 4?

[GeorgeCo]

I don't understand why this is tagged blocked: info-needed. Can someone explain what info is needed or remove the tag? I'm the filer of 6342 which is marked a duplicate of this. BTW this appears as a regression to me since cabal v1-install works, see 6342

[fgaz]

So I was thinking about this, specifically about how to store the dependencies/targets and both the two possible file-based solutions have downsides:

• storing it centrally needs gc and can cause problems if the user manually edits an env file
• storing it in a separate file in the same directory as the env file is just messy -- I already have enough config files

But then I realized there's a third way, which may have been obvious, but I don't see any mention of it in this thread so I'm writing it down for reference:

• store the info in a comment in the environment file (yes, they support comments too, fortunately).

Here's a sketch of what this would require:

• adding a CabalTargets [TargetSelector] (or similar) constructor to the GhcEnvironmentFileEntry datatype
• reading the comment before installing (and maybe falling back on the current behavior if the file is not cabal-generated) and using that as additional targets instead of using the env file contents
  • making cli targets with the same name override the ones from the file (for example semigroups-0.18.5 from cli overrides semigroups-0.18.4 from the comment)
    • being explicit about that: "Warining: Overwriting package x-x.y with x-X.Y"
• writing the (merged) targets to the comment when installing on an environment file (MAYBE only if it's a new env or if it's cabal-generated (=it already has the comment))
  • also writing a brief explanation ("the following comment contains...") in a comment
• printing lots of info about what cabal is doing ("adding targets to env file /home/foo/.ghc/...", "using targets x y and x from env file", etc)
  • especially on errors

[phadej]

From my experience with cabal-env, you not only want to store [TargetSelector] but also:

• explicitly hidden packages (e.g. to hide base-compat where you have base-compat-batteries)

I.e. you need a way to also hide some packages.

Which leads to design question: should only explicitly selected be visible? I think so. But then adding stuff (which is already there) should be fast: Then you'd need to store plan.json too, so you don't need to resolve every environment modification operation: if the new target is already there (transitively): just enable it.

EDIT: I realise if you only make specifically requested packages public, you don't need to store the list of hidden packages. But you still want a faster regeneration, when they are in plan.

[GeorgeCo]

@hvr You tagged this "blocked: info-needed" and then wrote "this needs to have a solution/algorithm/heuristic designed to proceed". I thought "blocked: info-needed" was reserved for issues that needed info to reproduce. If it is used for all bugs without a solution wouldn't there by many more tagged "blocked: info-needed"? In any case your comment is a year old now, it would be nice if somebody could design a solution/algorithm/heuristic to proceed. I'm surprised cabal 3.0 was released with this bug.

[phadej]

I removed the blocked: info-needed. The issue is well understood.

Software has bugs, some are fixed on time, some aren't. This is bad, but not critical, that the release could been postponed.

[GeorgeCo]

Thanks!!

On Thu, Nov 21, 2019 at 10:07 AM Oleg Grenrus [email protected] wrote:

I removed the blocked: info-needed. The issue is well understood.

Software has bugs, some are fixed on time, some aren't. This is bad, but not critical, that the release could been postponed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/haskell/cabal/issues/5559?email_source=notifications&email_token=ABQIJ642ACY7W6BA3RRC2U3QU2IYJA5CNFSM4FSZSMLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE2KWIY#issuecomment-557099811, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQIJ636QOXQ7NXEJMMKCDLQU2IYJANCNFSM4FSZSMLA .

[phadej]

I prototyped a variant along @fgaz comment in cabal-env (debug messages show how long the commands take to run)

---

Let's install some

[mymachine] ~ % cabal-env some
[   0.00033] debug: runProcess cwd=/home/phadej/.ghc ghc --info
[   0.04574] debug: GHC environment directory: /home/phadej/.ghc/x86_64-linux-8.6.5/environments
[   0.04589] debug: Generated fake-package.cabal
[   0.04652] debug: runProcessOutput cwd=/tmp/cabal-env-fake-package-XXXX-09028e4cdb6b7ff7 cabal v2-build all --builddir=dist-newstyle
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - fake-package-0 (lib) (first run)
Configuring library for fake-package-0..
Preprocessing library for fake-package-0..
Building library for fake-package-0..
[  13.45922] debug: writing environment file

We can check what's in the environment:

[mymachine] ~ % cabal-env --show
[   0.00022] debug: runProcess cwd=/home/phadej/.ghc ghc --info
array-0.5.3.0
base-4.12.0.0
deepseq-1.4.4.0
fake-package-0
ghc-prim-0.5.3
integer-gmp-1.0.2.0
rts-1.0
some-1.0.1

Next we can try to install some again, and also specify that only targets are exposed:

[mymachine] ~ % cabal-env some --no-transitive
[   0.00041] debug: runProcess cwd=/home/phadej/.ghc ghc --info
[   0.04631] debug: GHC environment directory: /home/phadej/.ghc/x86_64-linux-8.6.5/environments
[   0.04794] debug: Everything in plan, regenerating environment file
[   0.04836] debug: writing environment file

[mymachine] ~ % cabal-env --show              
[   0.00022] debug: runProcess cwd=/home/phadej/.ghc ghc --info
array-0.5.3.0 (hidden)
base-4.12.0.0
deepseq-1.4.4.0 (hidden)
fake-package-0 (hidden)
ghc-prim-0.5.3 (hidden)
integer-gmp-1.0.2.0 (hidden)
rts-1.0 (hidden)
some-1.0.1

Let's install another package, semialign doesn't depend on some, but both are in the environment:

[mymachine] ~ % cabal-env semialign
[   0.00034] debug: runProcess cwd=/home/phadej/.ghc ghc --info
[   0.04533] debug: GHC environment directory: /home/phadej/.ghc/x86_64-linux-8.6.5/environments
[   0.04687] debug: Generated fake-package.cabal
[   0.04765] debug: runProcessOutput cwd=/tmp/cabal-env-fake-package-XXXX-e1a015828dedacba cabal v2-build all --builddir=dist-newstyle
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - fake-package-0 (lib) (first run)
Configuring library for fake-package-0..
Preprocessing library for fake-package-0..
Building library for fake-package-0..
[  13.79145] debug: writing environment file

[mymachine] ~ % cabal-env --show   
[   0.00022] debug: runProcess cwd=/home/phadej/.ghc ghc --info
Cabal-2.4.0.1 (hidden)
QuickCheck-2.13.2 (hidden)
StateVar-1.2 (hidden)
aeson-1.4.6.0 (hidden)
array-0.5.3.0 (hidden)
assoc-1.0.1 (hidden)
attoparsec-0.13.2.3 (hidden)
base-4.12.0.0
base-compat-0.11.0 (hidden)
base-orphans-0.8.1 (hidden)
bifunctors-5.5.5 (hidden)
binary-0.8.6.0 (hidden)
bytestring-0.10.8.2 (hidden)
cabal-doctest-1.0.8 (hidden)
comonad-5.0.5 (hidden)
containers-0.6.0.1 (hidden)
contravariant-1.5.2 (hidden)
deepseq-1.4.4.0 (hidden)
directory-1.3.3.0 (hidden)
distributive-0.6.1 (hidden)
dlist-0.8.0.7 (hidden)
fake-package-0 (hidden)
filepath-1.4.2.1 (hidden)
ghc-boot-th-8.6.5 (hidden)
ghc-prim-0.5.3 (hidden)
hashable-1.3.0.0 (hidden)
integer-gmp-1.0.2.0 (hidden)
integer-logarithms-1.0.3 (hidden)
mtl-2.2.2 (hidden)
parsec-3.1.13.0 (hidden)
pretty-1.1.3.6 (hidden)
primitive-0.7.0.0 (hidden)
process-1.6.5.0 (hidden)
random-1.1 (hidden)
rts-1.0 (hidden)
scientific-0.3.6.2 (hidden)
semialign-1.1
semigroupoids-5.3.3 (hidden)
some-1.0.1
splitmix-0.0.3 (hidden)
stm-2.5.0.0 (hidden)
tagged-0.8.6 (hidden)
template-haskell-2.14.0.0 (hidden)
text-1.2.3.1 (hidden)
th-abstraction-0.3.1.0 (hidden)
these-1.0.1 (hidden)
time-1.8.0.2 (hidden)
time-compat-1.9.2.2 (hidden)
transformers-0.5.6.2 (hidden)
transformers-compat-0.6.5 (hidden)
unix-2.7.2.2 (hidden)
unordered-containers-0.2.10.0 (hidden)
uuid-types-1.0.3 (hidden)
vector-0.12.0.3 (hidden)

We realise that we also need these to use with semialign, let's quickly add it:

[mymachine] ~ % cabal-env these 
[   0.00034] debug: runProcess cwd=/home/phadej/.ghc ghc --info
[   0.04744] debug: GHC environment directory: /home/phadej/.ghc/x86_64-linux-8.6.5/environments
[   0.05555] debug: Everything in plan, regenerating environment file
[   0.05585] debug: writing environment file

And if we look at the environment file, it's what we'd expect there to be. Additional state is stored in the comments, with a last plan.json (lzma'd and base64 encoded)

[mymachine] ~ % head -n 50 ~/.ghc/x86_64-linux-8.6.5/environments/default 
-- This is GHC environment file written by cabal-env
--
clear-package-db
global-package-db
package-db /cabal/store/ghc-8.6.5/package.db
package-id base-4.12.0.0
package-id semialign-1.1-84da55bbf2b8473378fc710785d028724b407bace1d0b6a5adf9796bfd9b8e24
package-id some-1.0.1-3f8bfae892e7b2045f6749d5ebfe20ba131cda9ec4c978be8daa71fd6de6d01f
package-id these-1.0.1-a6977f7f9e17b9eb6d5b99dad0e10a320cca7a01824ae72489249dcee528152c
-- cabal-env packages:
-- cabal-env     semialign -any,
-- cabal-env     some -any,
-- cabal-env     these -any
-- cabal-env 
-- cabal-env plan:
-- cabal-env     /Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4GzYFehdAD2IiGZT0UYdkKRpnoLIXvjSQtxNmBIrpb5YIES6
-- cabal-env     JkSZqIZoECvueNm4YhLHwFaGdvq81ggXmJko+t9C8tOJhIKhFHo8lTO2sqk0hhHuAfHYthXGZFux+Ixy
-- cabal-env     smLg7lu8Mn49kPpAyr0l8ppWel+Ram7h7ZFbf7MD9X08gw0HuJ92MKiR7NdQPO4r1MtOciylc7mCdfmB
-- cabal-env     qV+tw++SYB/1LIpbOkaPO4Z+Dv/bJi6mhw2uIgIqfqwbIYKH5NlUqsQ4bH7/tlkiLdAaZwDfcmAZixNc
-- cabal-env     wBafOZvaArYKWU1Dk3nrcOifm3Q8DVvd2HCPVNco3nH0zldLwqMaPulgrSpAkpBSbPDmBb5eZqWEsEiq
-- cabal-env     azaWXd3YV4hq8XJI7iinaRGsU8b9lW3oIPA+5GtdJKqEdnq++/hkoReyewyGpSxWDQrNzmjSNs665Fcj
-- cabal-env     MplklRmd7uTpgsIO5pdtaHHkc7/QQtU49PSsCUDvL/YKW1YMsCJfV2KXN3fHuwNb1aoFClxC6EfZ535a
-- cabal-env     CaXQ0k0buIDitmRkXYXFllCSdiS8Ud5KtdLW1IcXlpOoMxfZ0OdntCR6/RaFzocksBa75ATwoZrNt1Pf
-- cabal-env     iN9kYDZ57io5Ieo91SROmP66QL/zteB5FqWSGU9Ra7pBG5MvH22pjNlqlekEC9E/EERWznTOOt2CacMP
-- cabal-env     baBUwZO8x0iCvwZtUv2qxYkjLdF2FlNEyxfkcVQ3uZ8mhR2wE+Ea8jMCUpPmQiMkpNc/NKeRGOaf3q5L
-- cabal-env     5XOPMoLLNkQBIpFtIPsmhX3lwMLngx1qAp/LhgTGVMJEfbZOaN2GU6jH/1LPr5MFANQzY5VOlrRctGAw
-- cabal-env     9ufLHUrW0gVt/7354LBjSY60VXtq7KmAEV/Gx1jkKinyIwmEfSXyCz3Dd1nTHcZkHZqKye/Vb29lvYgQ
...