Cabal cannot upload hs-source-dirs that only have autogen-modules

[Gabriella439]

Describe the bug

If a Cabal package has a source directory listed in hs-source-dirs that only contains autogen-modules then the package cannot be uploaded without a workaround. Specifically: cabal sdist will omit the entire source directory when creating the sdist and then cabal upload and/or Hackage rejects the package because the sdist is missing the source directory listed in hs-source-dirs.

To Reproduce

Steps to reproduce the behavior:

$ cat > autogen-issue.cabal <<EOF
name:                autogen-issue
version:             1.0.0
build-type:          Simple
cabal-version:       >=1.10

library
  build-depends:       base >=4.13 && <4.14
  hs-source-dirs:      src
  other-modules:       Example
  autogen-modules:     Example
  default-language:    Haskell2010
EOF

$ mkdir src

$ cat > ./src/Example.hs <<EOF
module Example where

main :: IO ()
main = mempty
EOF

$ cabal v2-sdist --list-only
./Setup.hs
./autogen-issue.cabal

… and if you were to try to upload the corresponding sdist to Hackage it would reject the upload, saying that the src directory is missing from the sdist.

If you delete the autogen-modules line then v2-sdist incorporates the directory correctly:

$ cabal v2-sdist --list-only
./Setup.hs
./autogen-issue.cabal
./src/Example.hs

Expected behavior

I expect cabal v2-sdist to incorporate all hs-source-dirs into the sdist (even if they are only incorporated as empty directories).

System information

• Operating system: macOS
• Cabal version: 3.2.0.0
• GHC version: 8.8.4

[phadej]

Why autogen-modules are generated in-tree. That doesn't seem right. AFAICS sdist works right here.

[Gabriella439]

@phadej: In the real example that motivated this we actually have two hs-source-dirs: src and gen. src has our normal source code and gen has generated code. Here is the original configuration that motivated this issue:

https://github.com/awakesecurity/proto3-suite/blob/489d5a3450411ebb9cf0ce7c0854b1c3f2f9eb69/proto3-suite.cabal#L107-L113

You still run into the same problem because gen is not bundled in the sdist because it only contains autogen-modules

[phadej]

I still don't understand.

How (e.g.) TestProto module is generated in proto3-suite example? The package is build-type: Simple?

You can generate autogen-modules only with build-type: Custom, see

• https://hackage.haskell.org/package/Cabal-3.4.0.0/docs/Distribution-Simple-BuildPaths.html#v:autogenPackageModulesDir and https://hackage.haskell.org/package/Cabal-3.4.0.0/docs/Distribution-Simple-BuildPaths.html#v:autogenComponentModulesDir

I suspect autogen-modules in a way which is not really supported. (The main use case is Paths_pkgname, and then custom stuff used by e.g. gi-gtk)

[Gabriella439]

@phadej: We generate them out-of-band (using Nix), but include them with the uploaded sdist. Are you saying that if we do that then it's more idiomatic to not classify them as autogen-modules?

[phadej]

If the module source files are in the sdist, then they are not autogen-modules (from Cabal perspective they are not generated - it doesn't care whether it's a human developer or some presdist script creating the source file).

[Gabriella439]

Alright, then I'll close this then. This is us misunderstanding autogen-modules

[fumieval]

@phadej If the autogen-modules section is missing, I'm unable to build it because cabal-install fails with the following error (cf. https://github.com/awakesecurity/proto3-suite/pull/185)

cabal: sdist of proto3-suite-0.5.0: Error: Could not find module: TestProto
with any suffix:
["gc","chs","hsc","x","y","ly","cpphs","hs","lhs","hsig","lhsig"]. If the
module is autogenerated it should be added to 'autogen-modules'.

[fumieval]

@phadej Any change you could consider lifting the restriction? I'm not sure what exactly goes wrong if such package is uploaded to Hackage

[Mikolaj]

@fumieval: hi! I lack context. Would you mind opening a new ticket with a repro? E.g., what happens if you add the modules to autogen-modules, what cabal-version you are using, etc.? If we are to permit a class of currently forbidden packages to Hackage, we need a ticket for a discussion involving Hackage maintainers, etc.

[fumieval]

@Mikolaj hi, sorry for late response. I made a small reproducer: https://github.com/fumieval/cabal-autogen-modules-issue

If a package lacks autogen-modules, cabal is unable to use the package as source-repository-package. Specifying autogen-modules fixes this problem, but doing so makes the package invalid (i.e rejected by Hackage).

[Mikolaj]

Thank you for the repro. Is it rejected by cabal check and/or cabal sdist as well? cc @ffaf1

A cheap guess: would bumping cabal-version in .cabal file help? (but the error messages in the repro would probably suggest that; I haven't run the repro).

[fumieval]

Updating cabal-version didn't help. I made a smaller reproducing example:

https://github.com/fumieval/cabal-autogen-modules-issue/blob/98905bad79d3e25ce765019b791a837e25f690ea/target/autogen-issue-fumieval.cabal

cabal check and cabal sdist pass, however, Hackage rejects the package:

cabal-install version 3.6.2.0
compiled using version 3.6.2.0 of the Cabal library 
fumieval@Retrofractum target % cabal check
No errors or warnings could be found in the package.
fumieval@Retrofractum target % cabal sdist
Wrote tarball sdist to
/Users/fumieval/herp/hs/cabal-autogen-modules-issue/target/dist-newstyle/sdist/autogen-issue-fumieval-1.0.0.tar.gz
fumieval@Retrofractum target % cabal sdist --list-only
./autogen-issue-fumieval.cabal

fumieval@Retrofractum target % cabal upload dist-newstyle/sdist/autogen-issue-fumieval-1.0.0.tar.gz
hackage.haskell.org username: FumiakiKinoshita
hackage.haskell.org password: 
Uploading dist-newstyle/sdist/autogen-issue-fumieval-1.0.0.tar.gz...
Error uploading dist-newstyle/sdist/autogen-issue-fumieval-1.0.0.tar.gz: http
code 400
Error: Invalid package

'hs-source-dirs: src' directory does not exist.

I found a workaround (adding an empty file as one of extra-source-files) so I'm happy as long as the PR gets merged, but cabal check still has a false negative bug.

[Mikolaj]

[Apologies, wrong window. :D I've deleted the wrong comment.]

[gbaz]

I think this is an issue with sdist, and we should fix it. It should sdist even an empty directory assuming that the directory is listed in hs-source-dirs.

[ivanperez-keera]

Hitting this bug with a project that I'm trying to put on hackage that is mainly a language library automatically generated from a BNFC grammar.

[ivanperez-keera]

I'm working around the issue by creating .keep empty files and adding them to extra-source-files.