testsuite: skipifVersion "9.6.3" in VersionPriority tests looks overfitted

[mpickering]

The VersionPriority tests have code along the lines of

8832b2689a (Phil de Joux 2023-12-31 08:52:34 -0500 11)     skipIfGhcVersion "== 9.6.3"

but the reason seems to be an unrelated bug in the testsuite:

8832b2689a (Phil de Joux 2023-12-31 08:52:34 -0500  8)     -- To avoid this diff:
8832b2689a (Phil de Joux 2023-12-31 08:52:34 -0500  9)     --   -Build profile: -w ghc-9.6.3 -O1
8832b2689a (Phil de Joux 2023-12-31 08:52:34 -0500 10)     --   +Build profile: -w ghc-<GHCVER> -O1

This kind of bug happens in situations where you build cabal-tests with a different ghc version to the one on your path I think, in any case, that is the bug which should be fixed rather than workarounds which only apply to very specific cases in the testsuite.

cc @philderbeast

[philderbeast]

Could this be needed because the test suite output substitution is behaving differently when importing a stackage.org/.../cabal.config that sets the ghc version?

-- Stackage snapshot from: http://www.stackage.org/snapshot/nightly-2023-12-07
-- Please place this file next to your .cabal file as cabal.config
-- To only use tested packages, uncomment the following line:
-- remote-repo: stackage-nightly-2023-12-07:http://www.stackage.org/nightly-2023-12-07
with-compiler: ghc-9.6.3
...

[mpickering]

It seems that this test also depends on network access? Can we refactor it to avoid that? It fails locally on my machine which doesn't have network access.

[philderbeast]

We could but then import: https://... is an important thing to test too.

[philderbeast]

Can we refactor it to avoid that?

We could, the cabal.config we reference on stackage.org could be saved locally.

[philderbeast]

@mpickering it took me a while to reproduce but the circumstances that lead to the .out file diff are these;

$ git diff cabal-testsuite/PackageTests/VersionPriority/1-web.test.hs
...
 testVersionWin project =
     withProjectFile project $ do
-        fails $ cabal "v2-build" ["--dry-run"]
+        fails $ cabal "v2-build" []
 
 main = cabalTest . withRepo "repo" $ do
-    -- To avoid this diff:
-    --   -Build profile: -w ghc-9.6.3 -O1
-    --   +Build profile: -w ghc-<GHCVER> -O1
-    skipIfGhcVersion "== 9.6.3"
     testVersionWin "1-web-constraints-import.project"
     testVersionWin "1-web-import-constraints.project"

$ git diff cabal-testsuite/PackageTests/VersionPriority/1-web-constraints-import.project
...
 packages: .
-allow-newer: hashable:*
-constraints: hashable ==1.4.2.0
 
 -- NOTE: We could have grabbed this config from stackage but we don't to avoid
 -- making an HTTP request with the test. So instead we save it locally.
 --
 -- $ curl https://www.stackage.org/nightly-2023-12-07/cabal.config --output project-stackage/nightly-2023-12-07.config
 import: project-stackage/nightly-2023-12-07.config
-
-import: with-ghc.config

The .out changes depending on what the system ghc is set to. Because I've altered the .out so much, I'll show you the diff between subsequent runs;

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 9.8.1

$ cabal run cabal-testsuite:cabal-tests -- --with-cabal=./dist-newstyle/build/x86_64-linux/ghc-9.6.3/cabal-install-3.11.0.0/x/cabal/build/cabal/cabal cabal-testsuite/PackageTests/VersionPriority/1-web.test.hs --accept
...

$ cabal run cabal-testsuite:cabal-tests -- --with-cabal=./dist-newstyle/build/x86_64-linux/ghc-9.8.1/cabal-install-3.11.0.0/x/cabal/build/cabal/cabal cabal-testsuite/PackageTests/VersionPriority/1-web.test.hs --accept
...

$ cat cabal-testsuite/PackageTests/VersionPriority/1-web.out
# cabal v2-update
Downloading the latest package list from test-local-repo
# cabal v2-build
Resolving dependencies...
Build profile: -w ghc-9.6.3 -O1
...

Now doing the same but with a different system ghc version;

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 9.6.3

$ cabal run cabal-testsuite:cabal-tests -- --with-cabal=./dist-newstyle/build/x86_64-linux/ghc-9.6.3/cabal-install-3.11.0.0/x/cabal/build/cabal/cabal cabal-testsuite/PackageTests/VersionPriority/1-web.test.hs --accept
...
-Build profile: -w ghc-9.6.3 -O1
+Build profile: -w ghc-<GHCVER> -O1
...

$ cabal run cabal-testsuite:cabal-tests -- --with-cabal=./dist-newstyle/build/x86_64-linux/ghc-9.8.1/cabal-install-3.11.0.0/x/cabal/build/cabal/cabal cabal-testsuite/PackageTests/VersionPriority/1-web.test.hs --accept

$ cat cabal-testsuite/PackageTests/VersionPriority/1-web.out
# cabal v2-update
Downloading the latest package list from test-local-repo
# cabal v2-build
Resolving dependencies...
Build profile: -w ghc-<GHCVER> -O1
...

[philderbeast]

This is a bit of a gotcha with our testing setup.