New Cabal-syntax and an ancient Cabal conflict in a single package (e.g., old hackage-repo-tool on Hackage used both a new Cabal-syntax and ancient Cabal)

[Mikolaj]

See https://github.com/haskell/hackage-security/runs/7814984300?check_suite_focus=true

tests/TestSuite/InMemRepo.hs:17:1: error:
    Ambiguous module name ‘Distribution.Text’:
      it was found in multiple packages:
      Cabal-3.6.0.0 Cabal-syntax-3.8.1.0

Edit: this was first spotted in cabal CI, but the link I gave here was wrong or no longer works, so I've changed it.

[Mikolaj]

hackage-repo-tool depends on hackage-security, which perhaps is compiled with old Cabal bundled with GHC, but then for some reason (no idea why; perhaps something unrelated fails in the build plan and the solver flips the Cabal-syntax flag and then indeed new Cabal-syntax is mandatory) a new Cabal-syntax is picked and hence the conflict.

[Mikolaj]

Actually, hackage-repo-tool on Hackage is ancient, so this may be why it's broken. Let me update on Hackage.

[ulysses4ever]

CI link in the OP doesn’t show me anything meaningful (eg no error).

[Mikolaj]

CI link in the OP doesn’t show me anything meaningful (eg no error).

The never-ending yellow CI job if full of errors. :)

[ulysses4ever]

Oh, right, I see that.

[Mikolaj]

hackage-repo-tool on Hackage is ancient, but it has reasonable bounds: Cabal (>=1.14 && <1.26 || >=2.0 && <2.6 || >=3.0 && <3.4) and up to date hackage-security bounds. How come this leads to a clash of Cabal-1.24.2.0 and Cabal-syntax-3.8.1.0? I'm worried that user packages may be similarly affected.

[Mikolaj]

I'm changing the topic, because hackage-repo-tool will be updated on Hackage as soon as I get the authorization, but the problem is more general --- our shrewd scheme of bounds on Cabal-syntax, etc., probably doesn't quite work.

[ulysses4ever]

For hackage-repo-tool specifically: somehow, hackage-security-0.6.2.2 gets dependent on Cabal-syntax-3.8.1.0, despite this dependency being hidden behind a flag (cf. cabal-plan tred output below). Curious...

[Mikolaj]

The solver is permitted to flip (some) flags. I'm not an expert on that, though. I think the manual field determines that. If it's not manual, it's solver-amenable. I think in this case it was the intention that solver would flip the flag if newer Cabal can be used. But we haven't imagined a situation like that, where due to other bounds, the solver can't use a new Cabal in some of the deps, while it can in others.

[Mikolaj]

I think here is the main sketch of the scheme we adopted: https://github.com/haskell/hackage-security/pull/254#discussion_r742240938

[Mikolaj]

I think the plan might have been to also retro-actively revise bounds in old cabals, but I can't find any write-up about that and, if I'm not completely making this up, we probably forgot to do it.

[jneira]

The problem was noted here: https://github.com/haskell/cabal/issues/7974#issuecomment-1040860719

@gbaz and me suggested change bounds retroactively in the same issue https://github.com/haskell/cabal/issues/7974#issuecomment-1050199037 (not sure if that what are you remembering?)

but finally re-export all modules was the chosen solution

rereading the issue could give us more insights

[Mikolaj]

Given that this surfaced in the wild again, in https://github.com/haskell/hackage-server/issues/1130, I'm bumping priority. This is potentially a ticking bomb, though so far the ticks are sparse and people manage to work around faster than it counts down.

[andreasabel]

We can add cabal-install-3.6.2.0 to the list of victims of this bug (see https://github.com/haskell/cabal/pull/8435#issuecomment-1231911041).

[ulysses4ever]

FTR, Stack hit it and worked around by changing the source code to use package imports (which eliminates ambiguity): https://github.com/commercialhaskell/stack/pull/5886

[ulysses4ever]

Due to this issue I can't bisect a bug between cabal-install-3.6 and cabal-install-3.8: as noted above, 3.6 fails to build today due to something like:

      Expected type: Cabal-syntax-3.10.1.0:Distribution.Types.PackageId.PackageIdentifier
        Actual type: PackageId

Any workaround that doesn't require touching checked in files? E.g. can I add a cabal.project.magic that would allow to build cabal-install-3.6 out of a fresh checkout? What would be the minimal such cabal.project.magic? Just pin Hackage index to pre-Cabal-syntax and import cabal.project? Or maybe put some constraint in it (what)?

---

Update: I got around it and built cabal-install-3.6 by using the following cabal.project.magic

packages: Cabal/ cabal-testsuite/
packages: cabal-install/cabal-install-solver/
packages: cabal-install/
packages: solver-benchmarks/
packages: Cabal-QuickCheck/
packages: Cabal-tree-diff/
packages: Cabal-described
packages: Cabal-tests/
packages: cabal-benchmarks/

index-state: 2022-07-24T17:47:48Z

The date is a random point before Cabal-syntax was released on Hackage in August 2022.

[Bodigrim]

Is it the same issue @vdukhovni hit in https://mail.haskell.org/pipermail/ghc-devs/2023-November/021459.html?

[vdukhovni]

Is it the same issue @vdukhovni hit in https://mail.haskell.org/pipermail/ghc-devs/2023-November/021459.html?

Essentially, yes. Only in my case once the flag is flipped, I see a somewhat different conflict, but the effect is the same, old Cabal conflicting with new Cabal-syntax, which is selected when the flag flips in hackage-security. This stops being a problem with cabal-install 3.8 or later, but I had to build 3.0, 3.2, 3.4 and 3.6 from git repo with manual edits to a cabal.project.freeze file.

The current cabal-syntax flag is too eager too flip to on, even when building old cabal-install that is not compatible with new Cabal-syntax.

[andreabedini]

Not sure whether it is the same issue but I have recently noticed that hackage-security moved to using Cabal-syntax in a minor version bump, causing some planning issues:

https://github.com/haskell/cabal/pull/9457#issuecomment-1817384809

PS: I checked the email thread, yes, it is the same issue.

[ulysses4ever]

Maybe we should have posted a clarification on the mailing thread for posterity… Unfortunately, no one really knows a good solution it seems. The one I use is to freeze the index to a date in 2022-07.