haskell
Cabal ignores `hs-source-dirs:` if `main-is:` is a C source file
Describe the bug
After GHC 9.8.2/Cabal-3.10.2.0, and by GHC 9.10.1/Cabal-3.12.0.0, Cabal (the library) ignores hs-source-dirs if main-is is a C source file. Previously, Cabal respected hs-source-dirs.
To Reproduce
Imagine a small package. In directory app it has Main.hs with contents:
module Main ( main ) where
import Lib ( myMax )
main :: IO ()
main = print $ myMax 10 100
In directory src it has Lib.hs with contents:
module Lib ( myMax ) where
myMax :: Int -> Int -> Int
myMax x1 x2 = if x1 > x2 then x1 else x2
foreign export ccall myMax :: Int -> Int -> Int
In directory c-app it has main.c with contents:
#include <stdio.h>
#include <HsFFI.h>
#ifdef __GLASGOW_HASKELL__
#include "Lib_stub.h"
#endif
int main(int argc, char *argv[]) {
hs_init(&argc, &argv);
printf("%lld\n", myMax(10,100));
hs_exit();
return 0;
}
It has the usual Setup.hs:
import Distribution.Simple
main = defaultMain
and it has a Cabal file:
cabal-version: 1.12
name: haskell-c-tests
version: 0.1.0.0
build-type: Simple
library
exposed-modules: Lib
hs-source-dirs: src
ghc-options: -stubdir autogen-stubs
build-depends: base
default-language: Haskell2010
executable c-exe
main-is: main.c
hs-source-dirs: c-app
ghc-options: -no-hs-main
include-dirs: autogen-stubs
build-depends: base, haskell-c-tests
default-language: Haskell2010
executable haskell-exe
main-is: Main.hs
hs-source-dirs: app
ghc-options: -threaded -rtsopts -with-rtsopts=-N
build-depends: base, haskell-c-tests
default-language: Haskell2010
With GHC 9.8.2/Cabal-3.10.2.0 all is fine (extracts only):
❯ stack --stack-yaml stack-ghc-9.8.2.yaml runghc --no-ghc-package-path -- Setup.hs configure
Configuring haskell-c-tests-0.1.0.0...
❯ stack --stack-yaml stack-ghc-9.8.2.yaml runghc --no-ghc-package-path -- Setup.hs build -v
...
Preprocessing executable 'c-exe' for haskell-c-tests-0.1.0.0..
Building executable 'c-exe' for haskell-c-tests-0.1.0.0..
creating dist\build\c-exe
creating dist\build\c-exe\c-exe-tmp
Building C Sources...
creating dist\build\c-exe\c-exe-tmp
Running: "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.8.2\bin\ghc.exe" "-c" "-odir" "dist\build\c-exe\c-exe-tmp" "-Idist\build\c-exe\autogen" "-Idist\build\global-autogen" "-Idist\build\c-exe\c-exe-tmp" "-Iautogen-stubs" "-Idist\build\autogen-stubs" "-optc-O2" "-pgmc" "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.8.2\lib\../mingw/bin\clang.exe" "-hide-all-packages" "-no-user-package-db" "-package-db" "dist\package.conf.inplace" "-package-id" "base-4.19.1.0-6554" "-package-id" "haskell-c-tests-0.1.0.0-2Qfk7puHfWxBnTSX2kZfQz" "c-app\main.c" "-no-hs-main"
Linking...
Note that Cabal has passed "c-app\main.c" to GHC 9.8.2 (the second from last argument).
However, with GHC 9.10.1/Cabal-3.12.0.0, Cabal passes only "main.c" to GHC 9.10.1 and, consequently, fails to find main.c:
❯ stack --stack-yaml stack-ghc-9.10.1.yaml runghc --no-ghc-package-path -- Setup.hs configure
Configuring haskell-c-tests-0.1.0.0...
❯ stack --stack-yaml stack-ghc-9.10.1.yaml runghc --no-ghc-package-path -- Setup.hs build -v
...
Preprocessing executable 'c-exe' for haskell-c-tests-0.1.0.0...
Building executable 'c-exe' for haskell-c-tests-0.1.0.0...
creating dist\build\c-exe
creating dist\build\c-exe\c-exe-tmp
Wanted build ways: [StaticWay]
Building C Sources...
creating dist\build\c-exe\c-exe-tmp
Running: "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.1\bin\ghc.exe" "-c" "-odir" "dist\build\c-exe\c-exe-tmp" "-Idist\build\c-exe\autogen" "-Idist\build\global-autogen" "-Idist\build\c-exe\c-exe-tmp" "-Iautogen-stubs" "-Idist\build\autogen-stubs" "-optc-O2" "-pgmc" "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.1\lib/../mingw//bin\clang.exe" "-hide-all-packages" "-no-user-package-db" "-package-db" "dist\package.conf.inplace" "-package-id" "base-4.20.0.0-30dc" "-package-id" "haskell-c-tests-0.1.0.0-BiMnNkNFjFZ2oxYL44TYmr" "main.c" "-no-hs-main"
<command line>: does not exist: main.c
(stack --stack-yaml stack-ghc-9.10.1.yaml runghc --no-ghc-package-path -- etc runs runghc in the correct environment - that is, one that provides the relevant version of GHC and its Cabal boot package. The --no-ghc-package-path is because GHC_PACKAGE_PATH and Cabal are incompatible.)
Have you checked GHC 9.10/Cabal-3.10?
With Cabal-3.10.3.0, Stack-2.15.7, and GHC-9.10.1 the build seems to succeed.
With Cabal-3.12.1.0 and GHC-9.6.5 the build fails (Cabal cannot find the .c file).
Have you checked GHC 9.10/Cabal-3.10?
As the problem is what different versions of Cabal (the library) are, or are not, passing to GHC, I reason that changing GHC version for a given Cabal version has no effect. But I've not tested that empircially.
But I've not tested that empiricially
I have. The problem is with Cabal. :-(
I agree with @mouse07410. I was able to try the GHC 9.10.1/Cabal 3.10.2.0 combination by adding to the Cabal file:
build-type: Custom
custom-setup
setup-depends: base, Cabal ==3.10.2.0
and to stack-ghc-9.10.1.yaml:
snapshot: nightly-2024-04-04 # GHC 9.8.2
compiler: ghc-9.10.1
extra-deps:
- Cabal-3.10.2.0
- Cabal-syntax-3.10.2.0
# Specifying a boot package as an extra-dep requires its boot package dependencies to be
# specified also
- Win32-2.13.4.0@sha256:6a1299d051c514aa5ac7b77ef5f86be6c0aa6940b00302c6dc246192c7a97d99,5769
- binary-0.8.9.2@sha256:03381e511429c44d13990c6d76281c4fc2468371cede4fe684b0c98d9b7d5f5a,6611
- containers-0.6.8@sha256:bb2bec1bbc6b39a7c97cd95e056a5698ec45beb5d8feb6caae12af64e4bd823c,2670
- directory-1.3.8.5@sha256:fbeec9ec346e5272167f63dcb86af513b457a7b9fc36dc818e4c7b81608d612b,3166
- filepath-1.4.300.2@sha256:345cbb1afe414a09e47737e4d14cbd51891a734e67c0ef3d77a1439518bb81e8,5900
- parsec-3.1.17.0@sha256:8407cbd428d7f640a0fff8891bd2f7aca13cebe70a5e654856f8abec9a648b56,5149
- process-1.6.20.0@sha256:2a9393de33f18415fb8f4826957a87a94ffe8840ca8472a9b69dca6de45aca03,2790
- text-2.1.1@sha256:78c3fb91055d0607a80453327f087b9dc82168d41d0dca3ff410d21033b5e87d,10653
- time-1.12.2@sha256:88e8493d9130038d3b9968a2530a0900141cd3d938483c83dde56e12b875ebc8,6510
Actually, things are even more interesting. Below is a proof that the regression is in Cabal 3.12.0.0 library, with which Stack-2.15.7 is built.
- With Cabal-3.12.1.0, both
cabal buildandstack buildcommands fail. - With Cabal-3.10.3.0,
cabal build(which uses cabal 3.10 library) works, butstack build(which uses cabal 3.12 library) fails:
$ cabal build
Resolving dependencies...
Build profile: -w ghc-9.10.1 -O2
In order, the following will be built (use -v for more details):
- h-tst5-0.1.1.1 (lib) (configuration changed)
- h-tst5-0.1.1.1 (exe:h-tst5-exe) (configuration changed)
- h-tst5-0.1.1.1 (exe:h-tst5-c-exe) (configuration changed)
Configuring library for h-tst5-0.1.1.1..
Preprocessing library for h-tst5-0.1.1.1..
Building library for h-tst5-0.1.1.1..
Configuring executable 'h-tst5-exe' for h-tst5-0.1.1.1..
Configuring executable 'h-tst5-c-exe' for h-tst5-0.1.1.1..
Preprocessing executable 'h-tst5-c-exe' for h-tst5-0.1.1.1..
Building executable 'h-tst5-c-exe' for h-tst5-0.1.1.1..
Preprocessing executable 'h-tst5-exe' for h-tst5-0.1.1.1..
Building executable 'h-tst5-exe' for h-tst5-0.1.1.1..
[3 of 3] Linking /Users/ur20980/src/h-tst5/dist-newstyle/build/x86_64-osx/ghc-9.10.1/h-tst5-0.1.1.1/x/h-tst5-exe/opt/build/h-tst5-exe/h-tst5-exe [Library changed]
$ stack build
Warning: Stack has not been tested with GHC versions 9.10 and above, and using 9.10.1, this may fail.
Warning: Stack has not been tested with Cabal versions 3.12 and above, but version 3.12.0.0 was found, this may fail.
h-tst5> build (lib + exe) with ghc-9.10.1
Preprocessing library for h-tst5-0.1.1.1...
Building library for h-tst5-0.1.1.1...
Preprocessing executable 'h-tst5-c-exe' for h-tst5-0.1.1.1...
Building executable 'h-tst5-c-exe' for h-tst5-0.1.1.1...
<command line>: does not exist: test.c
Error: [S-7282]
Stack failed to execute the build plan.
While executing the build plan, Stack encountered the error:
[S-7011]
While building package h-tst5-0.1.1.1 (scroll up to its section to see the error) using:
/Users/ur20980/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_HwdwpEmb_3.12.0.0_ghc-9.10.1 --verbose=1 --builddir=.stack-work/dist/x86_64-osx/ghc-9.10.1 build lib:h-tst5 exe:h-tst5-c-exe exe:h-tst5-exe --ghc-options " -fdiagnostics-color=always"
Process exited with code: ExitFailure 1
$
@mouse07410, the official release of Stack 2.15.7 has a dependency on Cabal=3.10.3.0, but Stack does not use that dependency to build. For Simple it uses the boot package of the specified GHC. That can be overriden using a Custom build.
@mouse07410, the official release of Stack 2.15.7 has a dependency on
Cabal=3.10.3.0, but Stack does not use that dependency to build. ForSimpleit uses the boot package of the specified GHC. That can be overriden using aCustombuild.
Ok, in that case why does "cabal-install"-v3.10.3.0 build succeeds when "stack" build fails? Does GHC-9.10.1 use boot package with cabal-3.12?
Thanks everyone for helping to isolate the problem to Cabal-3.12. Next step would be bisect it, perhaps.
@mouse07410, yes GHC 9.10.1 comes with Cabal-3.12.0.0. This is a handy source: https://gitlab.haskell.org/ghc/ghc/-/wikis/commentary/libraries/version-history. This is another: https://www.snoyman.com/base/.
Thanks everyone for helping to isolate the problem to Cabal-3.12. Next step would be bisect it, perhaps.
Good idea, but infeasible for me, as I can't build Cabal locally. :-(
On isolating the code, my current assumption is that it is something to do with Distribution.Simple.GHC.Build.ExtraSources.buildSources:
buildCSources =
buildExtraSources
"C Sources"
Internal.componentCcGhcOptions
( \c -> do
let cFiles = cSources (componentBuildInfo c)
case c of
CExe exe
| let mainPath = getSymbolicPath $ modulePath exe
, isC mainPath ->
cFiles ++ [makeSymbolicPath mainPath]
-- NB: Main.hs is relative to hs-source-dirs, but Main.c
-- is relative to the package.
_otherwise -> cFiles
)
EDIT: That is, it seems (from the code comment above) to be a deliberate decision to change Cabal's design, but one that was undocumented.
EDIT2: The previous corresponding code appears to me to be Distribution.Simple.GHC.gbuildSources:
gbuildSources verbosity pkgId specVer tmpDir bm =
case bm of
GBuildExe exe -> exeSources exe
...
where
exeSources :: Executable -> IO BuildSources
exeSources exe@Executable{buildInfo = bnfo, modulePath = modPath} = do
main <- findFileEx verbosity (tmpDir : map getSymbolicPath (hsSourceDirs bnfo)) modPath
...
if isHaskell main || pkgId == fakePackageId
then
...
else let (csf, cxxsf)
| isCxx main = ( cSources bnfo, main : cxxSources bnfo)
-- if main is not a Haskell source
-- and main is not a C++ source
-- then we assume that it is a C source
| otherwise = (main : cSources bnfo, cxxSources bnfo)
in return BuildSources {
cSourcesFiles = csf,
cxxSourceFiles = cxxsf,
inputSourceFiles = [],
inputSourceModules = exeModules exe
}
EDIT3: Some commits of interest:
- 2decb0e729b795b198cdef719984ca4c5ca1f657 committed by @alt-romes
- 7b9058328e162a4cb707b5d5b25cd1d2df66680e committed by @sheaf (related PR https://github.com/haskell/cabal/pull/9718)
If the design choice is as intended, the 'problem' is that the change in the Package Description Format Specification was not documented in the following places:
- https://cabal.readthedocs.io/en/3.12/file-format-changelog.html#cabal-version-3-12
- https://cabal.readthedocs.io/en/3.12/cabal-package-description-file.html#pkg-field-executable-main-is
- https://cabal.readthedocs.io/en/3.12/cabal-package-description-file.html#pkg-field-hs-source-dirs
Thanks for investigating @mpilgrem. I'm fairly certain this change wasn't intended, and I think it should be fixed to regardless.
Thanks for your continued investigation @mpilgrem. I don't believe this change was intended.
My working directory patch didn't make it into Cabal 3.12, so I don't think that is the issue. I think 2decb0e looks like a possible culprit.
-- NB: Main.hs is relative to hs-source-dirs, but Main.c
-- is relative to the package.
I think this comment simply means that if main-is is a haskell file, it'll be looked up in hs-sources-dir(where haskell sources live). If it is a C file, it's looked up in the package root (ie not in the haskell sources dir).
I think this comment simply means that if
main-isis a haskell file, it'll be looked up inhs-sources-dir(where haskell sources live). If it is a C file, it's looked up in the package root (ie not in the haskell sources dir).
If hs-source-dirs: is not supposed to apply to non-haskell files (like main.c), IMHO there has to be a way to specify a directory other than the package root. Name it source-dirs: or whatever, but it has to be there.
I think this comment simply means that if
main-isis a haskell file, it'll be looked up inhs-sources-dir(where haskell sources live). If it is a C file, it's looked up in the package root (ie not in the haskell sources dir).If
hs-source-dirs:is not supposed to apply to non-haskell files (likemain.c), IMHO there has to be a way to specify a directory other than the package root. Name itsource-dirs:or whatever, but it has to be there.
Indeed. I've realised that I was being contradictory in saying we should fix this but asserting that C mains should not be found in hs-source-dirs, not realising that this is what the OP literally intended to do.
The patch me and @sheaf have put together restores the existing behaviour of looking for non-haskell mains in hs-source-dirs, as there's no clear advantage to breaking this existing behaviour. This regressed because of an oversight in the 2decb0e refactor.
being contradictory in saying we should fix this but asserting that C mains should not be found in
hs-source-dirs, not realising that this is what the OP literally intended to do.
What the OP (and I ;) need is a way to specify a directory, other than the package root, for non-Haskell files - but it does not have to be exactly hs- source-dirs.
If you replace hs-source-dirs: for non-Haskell files with something else - IMHO it would be OK, not risking to break a lot of existing packages. The main point of this issue (and of this problem) is not to lose this capability altogether (as the https://github.com/haskell/cabal/commit/2decb0e729b795b198cdef719984ca4c5ca1f657 apparently and unfortunately does).
What the OP (and I ;) need is a way to specify a directory, other than the package root, for non-Haskell files - but it does not have to be exactly hs- source-dirs.
Yes, indeed. But we also don't want to inadvertently break existing packages out there whose maintainers don't follow the latest in Cabal development ;), as this was previously the only way to set a source directory for a C main it seems.
I think it would be a welcome feature if someone implemented, say, non-hs-source-dirs, or other-source-dirs, ...., but it's not critical (I don't have the bandwidth to do this).
In the meantime, here's the fix which restores the existing behaviour, once again looking up C main files in hs-source-dirs: https://github.com/haskell/cabal/pull/10171
I first thought: "Umh, we broke something, we should fix it ASAP!" After reading more about it, I've started wondering: do we want to preserve this somewhat unintuitive behavior -- looking into hs-source-dirs for non-Haskell sources?
One way to answer this is to first ask: what would you choose if designing Cabal from scratch today? My feeling is that I'd not want that behavior.
Now, to the brakages. We probably don't want more brakages than necessary. But, first, we already caused breakage for people who relied on this behavior and are trying to update to cabal-install-3.12.1.0. And second, more importantly, do we understand how much of a breakage this causes?
In no way do I want to block #10171, just wanted to make sure that we make an informed choice with some reasoning under it (unlike the initial change that happened by accident).
The history is:
- March 2005: GHC 6.4 is released, introducing
Cabal-1.0. That documents:main-is: filename (required) The name of the source file containing the Main module, relative to the hs-source-dir directory. - September 2013:
Cabal-1.18.0is released. That documents:main-is: filename (required) The name of the .hs or .lhs file containing the Main module. Note that it is the .hs filename that must be listed, even if that file is generated using a preprocessor. The source file must be relative to one of the directories listed in hs-source-dirs. - August 2019:
Cabal-3.0.0.0is released. It documents - I think for the first time - that Cabal 1.18 had "Add[ed] support for specifying a C/C++/obj-C source file inexecutable:main-isfield." It also documentsmain-is: filename (required) The name of the .hs or .lhs file containing the Main module. Note that it is the .hs filename that must be listed, even if that file is generated using a preprocessor. The source file must be relative to one of the directories listed in hs-source-dirs. Further, while the name of the file may vary, the module itself must be named Main. Starting with cabal-version: 1.18 this field supports specifying a C, C++, or objC source file as the main entry point.
Thank you for the patch, @alt-romes and @geekosaur !
Would you have any idea when is it likely that we'd see a released version of Cabal with this patch/PR incorporated?
Thank you for the patch, @alt-romes and @geekosaur !
Would you have any idea when is it likely that we'd see a released version of Cabal with this patch/PR incorporated?
I'm not involved in cabal release management, but I would suggest to wait a little longer in case there are more regressions coming up, before cutting a new point release.
Otherwise you can always downgrade to 3.10.3.0.
That was our conclusion in our dev meeting yesterday, yes. There's also the complication that this is the Cabal library, and many things will stick to the version that ships with ghc (whether it should is another question) even if we make a minor release.
