Post-multilibs checklist

[fgaz]

#5526 (multiple public libraries) is in, but there are still a few things to add and polish

Visibility enforcement

• [x] Add a visibility field to sublibraries :fire: :fire: :fire: :exclamation: :exclamation: :exclamation:
  • [x] add the field
  • [x] update ghc-pkg
  • [x] populate the field
  • [x] enforce the field: #5848

cabal-install integration

• [x] Make the solver multilibs-aware #6039
  • [x] Enforce visibility at the solver level #6047 #6836
  • [x] Extend the compatibility to GHC<8.8 when using cabal-install #6836
  • [ ] Handle installed packages
• [ ] Update cabal init to take into account sublibraries when populating build-depends

Tests

• [x] Add tests
  • [x] positive (ghc>=8.8) (isssue #6409)
  • [x] negative (all ghcs) (done in #5848)

Documentation

• [x] Update changelog 4d2ca5255c859821cd34e6c52f626b6edc9610a3
  • [x] #6362 Also update the cabal spec changelog, which I forgot
• [x] #8364 Update the Cabal and cabal-install docs (can be adapted from my post)

Bugfixes

• [x] #6038 (cannot depend on public sublibrary when the package does not contain a main library too)
• [ ] #8095 same but with Setup.hs/Cabal and installed packages
• [ ] #7375 (same as #8095??)
• [x] #5846 (boundless dependency fails to parse)
• [x] #6083 (edge case with the old syntax)
• [x] mylib:{ } ^>= 42 syntax is accepted, though docs say it is not (#6083 related, makes sense to fix at the same time?) - fixedin 9b3686fb7c2bc1b7b9ea735d23f32cfe47c22353
• [x] #6281 (parse failure with mixins)
• [ ] #6358 Reconfiguration avoidance not affected by library visibility change
• [x] :fire: #7270 :fire: Cabal's simple solver may not pick up sublibraries unless --dependency is used?
  • [ ] Deprecate lookupDependency in favor of lookupInternalDependency and maybe a new lookupPackageDependency that looks up all libraries of a package

Ecosystem changes

• [ ] Make hackage render sublibraries too
  • [ ] haddock
  • [ ] cabal #5949
  • [x] hackage https://github.com/haskell/hackage-server/issues/1119
• [ ] Nix https://github.com/NixOS/cabal2nix/issues/539

[phadej]

I mark this as high priority as 3.0 cannot be released before Add a visibility field to sublibraries is done.

ping @23Skidoo @hvr

[phadej]

Should solver be able to flip flags based on visible field? I think it should, but I don't know if it will make implementation tricky.

I don't know a good motivation, maybe pkgA:foo is visible, only if pkgB:bar is available, and author of pkgA.

e.g. pkgA.cabal:

cabal-version: 3.0
name: pkgA
...

library
   -- available always
  build-depends: foo

flag foo
  description: Provide pkgA:foo
  manual: False
  default: True

library foo
  -- this is contrived example, but ...
  -- also it's somewhat fragile, as there isn't a complement constraint; but it should work
  if foo
    visible: True
    build-depends: pkgB:bar

 -- if not pkgB:bar, there's still some stuff to be used by pkgA

[phadej]

we will need MIN_VERSION_pkgB:foo kind of thing. No idea how the name should be generated though. Maybe double underscore: MIN_VERSION_pkgB__foo? as there cannot be package pkgB--foo?

[phadej]

Should visibility be recorded into InstalledInfo. Then ./Setup.hs will be able to fail if you try to "manually" use a internal library which was registered as non-visible.

Note: this change would need GHC support (as package database is ghc-pkg business), so this makes things tricky. (practically limiting multilibs for ghc-8.8+)

[fgaz]

re IPI: yeah, I came to the same conclusion but didn't know how to go at it :-/

Is it just a matter of updating the cabal submodule in the ghc repo?

[phadej]

@fgaz essentially yes.

[phadej]

I'm chaging the milestone to 3.2.

@fgaz are you confident you can do this by the mid December (only two weeks!), or should we be waiting this only for 3.4?

[phadej]

Moving to 3.4

[cartazio]

how can this be progressed? :)

[fgaz]

Hmm actually some of these may be done. Let me update it first

[fgaz]

@phadej thank you for all the work you've been doing on fixing those bugs, especially #6893!

[phadej]

@fgaz is anyone working on Handle installed packages and Make the solver multilibs-aware #6039?

If not, I'll stop waiting for them for Cabal-3.4?

---

Also if you could check what's missing for Also update the cabal spec changelog, which I forgot, I can do that as well if there are missing points.

[fgaz]

@phadej

is anyone working on Handle installed packages and Make the solver multilibs-aware

not me, maybe @grayjay? The solver is multilibs-aware now btw (I just checked the box to make it clearer). The remaining subtask is not super important, because with v2- there's almost always a source package from which visibility can be determined. The only counterexample I can think of is ghc-bundled packages, but those don't have sublibraries.

what's missing for Also update the cabal spec changelog

Well, there isn't any mention of the multilibs feature, despite it being implemented Cabal-side. Would it be ok to add it to the 3.0 changelog or is that frozen and the 3.4 one should be used?

[fgaz]

The only counterexample I can think of is ghc-bundled packages

Duh, there's also distros' packages :-/ Does anyone use those in combination with cabal-install v2- though? I can only think of nix

[phadej]

The remaining subtask is not super important, because with v2- there's almost always a source package from which visibility can be determined. The only counterexample I can think of is ghc-bundled packages, but those don't have sublibraries.

I don't agree. I have heard this argument before, and ended up fixing issues myself because someone run into them. I won't be spending any time arguing whether it's important or not. If it's not done and known issue, the feature is not ready.

The Nix(pkgs) folks how install packages into global package database will run into this issue.

[grayjay]

I haven't had time to look into handling installed packages.

[phadej]

@grayjay no worries, as long as it's still on your todo list, I'm happy :)

[recursion-ninja]

I've been waiting for about 5 months to upload an updated package to Hackage which was reworked to use multiple libraries.

Is there an ETA for Hackage supporting the multiple libraries feature?

[fgaz]

Afaik the warning and lack of installed-public-sublibs support in cabal-install does not prevent hackage to implement/enable multilibs support, since hackage deals with source packages. Not sure if they have plans for this, I don't see any issue about it in their tracker and https://github.com/haskell/hackage-server/issues/852 is closed

[fgaz]

~~#6470 is another one, Cabal-related this time :-/~~

[fgaz]

Handle installed packages

So I just tried to figure out exactly what problems arise from this, and my conclusion is... none? since v2- (and maybe with v1- too?), cabal-install only handles packages of which the package description is known (either from an index or local source), and there's no way to make it use a package database directly as far as I can see. So if packages from an existing package-db are only picked up when they are known from an index => the .cabal is known => the visibility is known => no visibility errors.

The only component that can deal with sourceless/indexless package dbs, as far as I can see, is Cabal. But then the "handle installed packages" issue does not apply, because it's cabal-install-solver-specific.

Am I missing some way of using indexless package dbs with cabal-install? @grayjay @phadej

[phadej]

So if packages from an existing package-db are only picked up when they are known from an index => the .cabal is known => the visibility is known => no visibility errors.

This is not true. Packages from existing package-db ARE picked even they aren't in any repository index. SImple example: no package repositories => you can still use base, transformers etc bundled with the GHC.

The last point is crucial for nixpkgs based usage, where nix setup populates the global package db (and it would be great if users disabled hackage.haskell.org repository so cabal-install won't try to reinstall stuff).

[fgaz]

Hmm, I must have done something wrong then. I tried to build a package with a public sublibrary with nix, then I tried to use it in another project and cabal would complain (unknown package: a (dependency of b)). When I added a file+noindex repo with a inside, it worked (without rebuilding a, so it used the one from nix).

[phadej]

In short, public sublibraries should work with ./Setup interface only. Without any reliance on cabal-install infrastructure.

What is still unclear to me is when visibility is enforced on that level and when not.

One solution would be to not enforce visibility in ./Setup at all and make all sublibraries public by default; and then make cabal-install "the bad guy" which would impose extra constraints.

Note: ./Setup acts as a sanity check. But e.g .--exact-configuration makes it to not check compatibility.

You have to keep in mind how this would fit into stack model too. They do use --exact-configuration so ./Setup won't check bounds. Having ./Setup not check visibility would be easy solution. Though then checking of that should be implemented in stack (at some point it would be required, I guess, but there won't be hard requirement).

[fgaz]

For the last part there's already a --allow-depending-on-private-libs Cabal flag

[fgaz]

Here's a more extensive reply:

In short, public sublibraries should work with ./Setup interface only. Without any reliance on cabal-install infrastructure.

This already works since #5526 was merged, at least with --dependency / --exact-configuration.

I just tested if it works between nix packages and it doesn't, probably because they don't explicitly use --dependency, and Cabal's simple solver can't pick up sublibs on its own (I'll have to check if I have to fix this. edit: yes I do: #7270)

---

What is still unclear to me is when visibility is enforced on that level and when not.

With #5848, visibility is stored in the IPI and it's always enforced at the Cabal/Setup.hs level.

We have a --allow-depending-on-private-libs Cabal/Setup.hs flag to allow jailbreaking for various reasons, like extending multilibs support to old ghc versions.

So this covers Setup.hs-managed -> Setup.hs-managed dependencies.

---

cabal-install-managed -> cabal-install-managed dependencies work, since there we do have access to the package description

---

cabal-install-managed -> Cabal/Setup.hs-managed

I think this worked before #6836: all libraries were treated as public and failure happened later in the Cabal stage, which was not ideal. After #6836: I'm not sure. It may work for ghc>8.8 or it may not work at all. I'm trying to check that, but I'm having trouble making cabal-install pick up the external package db. I already tried the --package-db flag, but it seems to have no effect:

> cabal build b --package-db ../a/dist-newstyle/packagedb/ghc-8.10.3 -v3
[...]
/nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3/bin/ghc --print-global-package-db
Reading installed packages...
/nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3/bin/ghc-pkg dump --global -v0
/nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3/bin/ghc --print-libdir
[...]
[__4] trying: ghc-prim-0.6.1/installed-0.6.1 (dependency of base)
[__5] unknown package: a (dependency of b)

edit: --package-db seems to be one of those inactive flags inherited from Cabal... how do I reproduce this then? I can't install packages in the global package db in nixos, so I guess I need a virtual machine or container :-/

[fgaz]

Well, it looks like installed packages are problematic in general: #7270 :-/

[kokobd]

Is this still being worked on?

[fgaz]

@kokobd yes... when contributors have the time. Missing items are mostly nice-to-haves and edge cases though. The ongoing stuff is:

• In the cabal-install solver, consider installed packages with multiple public libraries (last part of #6039)
• On the hackage side, haskell/hackage-server#1119

Help is appreciated of course!

[ulysses4ever]

Another item on that list in the OP, perhaps:

• https://github.com/haskell/cabal/issues/9122