cabal

cabal copied to clipboard

Describe the bug The --haddock-quickjump build flag does not apply when building Haddocks of dependencies. If specified, after building dependencies, Cabal fails with an internal error due to not finding a doc-index.json file in one of the dependencies. Upon checking the store, none of the built dependencies have a doc-index.json.

A workaround is to replace --haddock-quickjump with --haddock-options=--quickjump. Dependencies then have doc-index.jsons generated, and no error occurs.

To Reproduce Steps to reproduce the behavior:

$ cabal v2-build --enable-documentation --haddock-quickjump
Build profile: -w ghc-9.2.2 -O1
...
haddock: internal error: /home/raehik/.cabal/store/ghc-9.2.2/OneTuple-0.3.1-d4f76dedc5429e37aa1b698675999d94bfbe1d373c7cc55126ec565cbd4a8dd3/share/doc/html/doc-index.json: openBinaryFile: does not exist (No such file or directory)
cabal: Failed to build documentation for medlib-0.1.0.

I tested a handful of small projects with dependencies, and this issue occurs in the same way. (OneTuple seems to get picked a lot, probably due to being lexicographically early and having few dependencies of its own.)

Expected behavior

• Haddock configuration, specifically the --quickjump flag is applied to building dependencies.
• A missing doc-index.json doesn't cause an internal error (better error message? attempt to skip usage?).

System information

• Arch Linux
• Cabal 3.6.2.0, GHC 9.2.2 (default config, via ghcup)

Additional context The original command that was failing me for was cabal haddock --haddock-for-hackage --enable-documentation, which I have CI run to generate Haddock artifacts remotely. #7879 noted that --haddock-for-hackage implies --haddock-quickjump. I've been using that command for a while, and only recently it started failing. It might be related to a recent Cabal or GHC (or Haddock) release.

Here's a recent CI build log from a 1-file project of mine where the problem cropped up.

Seems like bug. Thank you.

OOI, do you feel like --haddock-quickjump adds value compared to --haddock-options=--quickjump?

OOI, do you feel like --haddock-quickjump adds value compared to --haddock-options=--quickjump?

Yes, mostly because --haddock-for-hackage implies --quickjump. Seems sensible to treat it specially.

@Kleidukos: do you know if anything around --quickjump has been changed lately in haddock? Or is it obvious this is a cabal bug? I assume the haddock in question is the newest on Hackage.

Not that I remember, but this very well might be something that others have hit but didn't report.

Since Cabal is the interface here, I'd be interested to know if it passes the appropriate flags correctly to every dependency. I must admit my knowledge of haddock-cabal interactions is fairly lacking.

I'm looking into this, I'll see what I can find.

I downloaded @raehik 's repo and tested on that two different commands: cabal v2-build --enable-documentation --haddock-quickjump and cabal haddock --haddock-quickjump. The build command errored out like is stated in the the beggining of the issue, and when I ran the haddock command. This was the output:

Resolving dependencies...
Build profile: -w ghc-9.2.2 -O1
In order, the following will be built (use -v for more details):
 - refined-with-0.3.0 (lib) (first run)
Configuring library for refined-with-0.3.0..
Preprocessing library for refined-with-0.3.0..
Running Haddock on library for refined-with-0.3.0..
Warning: The documentation for the following packages are not installed. No
links will be generated to these packages: OneTuple-0.3.1, QuickCheck-2.14.2,
StateVar-1.2.2, aeson-2.0.3.0, assoc-1.0.2, attoparsec-0.14.4,
attoparsec-0.14.4, base-compat-0.12.1, base-compat-batteries-0.12.1,
base-orphans-0.8.6, bifunctors-5.5.12, comonad-5.0.8, contravariant-1.5.5,
data-fix-0.3.2, distributive-0.6.2.1, dlist-1.0, hashable-1.4.0.2,
indexed-traversable-0.1.2, indexed-traversable-instances-0.1.1,
integer-logarithms-1.0.3.1, primitive-0.7.3.0, random-1.2.1.1, refined-0.6.3,
scientific-0.3.7.0, semialign-1.2.0.1, semigroupoids-5.3.7, splitmix-0.1.0.4,
strict-0.4.0.1, tagged-0.8.6.1, text-short-0.1.5, th-abstraction-0.4.3.0,
these-1.1.1.1, these-skinny-0.7.5, time-compat-1.9.6.1,
transformers-compat-0.7.1, unordered-containers-0.2.19.1, uuid-types-1.0.5,
vector-0.12.3.1, witherable-0.4.2
 100% ( 22 / 22) in 'Refined.WithRefine'
Warning: Refined.WithRefine: could not find link destinations for:

        - Data.Aeson.Types.FromJSON.FromJSON
        - Refined.Predicate
        - Data.Aeson.Types.FromJSON.parseJSON
        - Data.Aeson.Types.Internal.Value
        - Data.Aeson.Types.Internal.Parser
        - Data.Aeson.Types.FromJSON.parseJSONList
        - Data.Aeson.Types.ToJSON.ToJSON
        - Data.Aeson.Types.ToJSON.toJSON
        - Data.Aeson.Types.ToJSON.toEncoding
        - Data.Aeson.Encoding.Internal.Encoding
        - Data.Aeson.Types.ToJSON.toJSONList
        - Data.Aeson.Types.ToJSON.toEncodingList
        - Data.Hashable.Class.Hashable
        - Data.Hashable.Class.hashWithSalt
        - Data.Hashable.Class.hash
        - Refined.RefineException
        - Refined.Unsafe.Type.Refined
Documentation created:
/home/colton/tmp/refined-extra/dist-newstyle/build/x86_64-linux/ghc-9.2.2/refined-with-0.3.0/doc/html/refined-with/index.html

The command succedded but there were some dependencies that did not find documentation. I cloned haddock and patched the error to make it fail gracefully (saying "File not found" with a warning) and this was now the output of the build command:

Resolving dependencies...
Build profile: -w ghc-9.2.2 -O1
In order, the following will be built (use -v for more details):
 - refined-with-0.3.0 (lib) (first run)
Configuring library for refined-with-0.3.0..
"/home/colton/.ghcup/ghc/9.2.2/lib/ghc-9.2.2"
Preprocessing library for refined-with-0.3.0..
Building library for refined-with-0.3.0..
[1 of 2] Compiling Paths_refined_with ( /home/colton/tmp/refined-extra/dist-newstyle/build/x86_64-linux/ghc-9.2.2/refined-with-0.3.0/build/autogen/Paths_refined_with.hs, /home/colton/tmp/refined-extra/dist-newstyle/build/x86_64-linux/ghc-9.2.2/refined-with-0.3.0/build/Paths_refined_with.o, /home/colton/tmp/refined-extra/dist-newstyle/build/x86_64-linux/ghc-9.2.2/refined-with-0.3.0/build/Paths_refined_with.dyn_o )
[2 of 2] Compiling Refined.WithRefine ( src/Refined/WithRefine.hs, /home/colton/tmp/refined-extra/dist-newstyle/build/x86_64-linux/ghc-9.2.2/refined-with-0.3.0/build/Refined/WithRefine.o, /home/colton/tmp/refined-extra/dist-newstyle/build/x86_64-linux/ghc-9.2.2/refined-with-0.3.0/build/Refined/WithRefine.dyn_o )
Preprocessing library for refined-with-0.3.0..
Running Haddock on library for refined-with-0.3.0..
Warning: The documentation for the following packages are not installed. No
links will be generated to these packages: attoparsec-0.14.4
 100% ( 22 / 22) in 'Refined.WithRefine'
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/OneTuple-0.3.1-d4f76dedc5429e37aa1b698675999d94bfbe1d373c7cc55126ec565cbd4a8dd3/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/QuickCheck-2.14.2-edca112bb602504405d0748c8b21e894650fc09c93ad829089fb38ef3fc82629/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/StateVar-1.2.2-ff637fbecfffde85b1380c9086f8c3c49de49ebbea7978b7d8dcf25a510781e4/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/aeson-2.0.3.0-f4023e7b2f80d12c102194df98fe277987b3561f9987102521b73ebbbbaaa464/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/assoc-1.0.2-ff32c250fcc96fb2d0f0f42e758dc9a9da35ababb6907dcff49bcdbf22c4ff64/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/attoparsec-0.14.4-4c60b4fa25cfaeed86dbde506709fed28bdedb6e6f542e1ef17fb48d7b1ff9b2/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/base-compat-0.12.1-069c50a7aa195259b05accad03ecd2ede280a523bda9ab68b1268e3311df8602/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/base-compat-batteries-0.12.1-76c04e71eeee9478e90cfae8157484f0f95cd4b5d4259895420eb4a665c347b2/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/base-orphans-0.8.6-2baf596496a50a207bc2260f9b9ecdc21b8f0ed1c50fa720ec542851f3e33f75/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/bifunctors-5.5.12-107d9fc6256587852db3a25c0ece4905978ca68617d0b1b9d7f089a134600bc9/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/comonad-5.0.8-83d0042a693ff84fd9d056662dd9d8804bae38deba1e2326578b336fe2542ec5/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/contravariant-1.5.5-b66553d970f91ea3ab320744f4c99dac2c5fee25e3af0cedc3f5afe5cb61e835/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/data-fix-0.3.2-a560822498d0d9e3eeb92ff0025ab07f399c16014445e2da345d1fb3a0d0c077/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/distributive-0.6.2.1-589cf878db1743f72e9658a1a8c3384735c75851b9f80e7d8af96238936fc73e/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/dlist-1.0-d9cf3817c7a7db843101da34c127e2413a122fa8148deb606dba610c1651c1e8/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/hashable-1.4.0.2-a4f5b19cd3cff5544d20705564fd8149c40da398efbf301d8e710f173f56a654/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/indexed-traversable-0.1.2-3611438ee7a4e689f511b138c1a086980067e1df60e4cde34ed3651224dfcae9/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/indexed-traversable-instances-0.1.1-af4556849df49fa7bf3ea7dcc90b78964143a631ca974fb2510c46cbf2fa9bec/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/integer-logarithms-1.0.3.1-71d18a11acfa27793627d47f70a166a92aacb116dfa7f1c2f0c68bb4d1a6cdcb/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/primitive-0.7.3.0-d5cb41566b1375cd2f96362e90e6efb805a1523d2938d68d599de3c1f3a4e7d3/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/random-1.2.1.1-dd2c5e17741947848ba7d3b9ae0615d6036ca66ddf69499faed9ff4f37e8fab1/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/refined-0.6.3-19c49f9f694961cb010d82ba5ee6d50a8925494d066c924f01cd0b08c03c6b91/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/scientific-0.3.7.0-c66d1053b67cb599ab77ec8241ad769ddcb1f41d676f426875896e805151b8de/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/semialign-1.2.0.1-d202600078bd25d2df619e696c97e3c9a7e34b332f1f6ec4b0e82cd605118492/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/semigroupoids-5.3.7-63a5611972eb14cb488537cb0244be53a8a621a1b9ec772592419ad55f866c47/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/splitmix-0.1.0.4-db3fdcd32289928774a2c84f42e0c4e32bdc9cf7d2e060a17725211169e4460a/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/strict-0.4.0.1-c605baaff253f2a46ffe323fcd8f3ccf5416e62713dae53b10cdfe5fee04ba18/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/tagged-0.8.6.1-27009b2e4fe1f9c7cbdc9bd44685fcbbd79d5afb6837e28248c7987974dcae64/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/text-short-0.1.5-0380526384eb4376055aacb7e7fd6af998c13805cdb45824c301037cd4f946ed/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/th-abstraction-0.4.3.0-23717d25a63f0448b4a474a90285c484e5230fb9664e10a51ca4815cf991f46f/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/these-1.1.1.1-4261e090a45b0d33dd243d2347f173cc33946f66687ec0ac6fd6c96bf773e07e/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/these-skinny-0.7.5-11811332c327f00497ee9a7518291945402dc5e651275fb8b3aeef2e06662b80/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/time-compat-1.9.6.1-370b0e9e4b5f426d3e4e6273fccc077f3b96e27f02d423539259c477eed6c2be/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/transformers-compat-0.7.1-53eaa2626fdc651c44f5f15f940b278f0c78bd53bd9a03042bb55641d79d6309/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/unordered-containers-0.2.19.1-ac153eb4bc2da45e8562aca94d656f1560401b2c8412e626538b9846bacf2338/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/uuid-types-1.0.5-771da564fd63a2e1d0023523b1cda94a14f5ec4193c92a36a004f8d8c8b833c1/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/vector-0.12.3.1-a072a953b5f1a2fd70462d2dde5345a1496198e1e18d8369fc01da0637e813a2/share/doc/html/doc-index.json: File not found
haddock: Coudn't parse /home/colton/.cabal/store/ghc-9.2.2/witherable-0.4.2-720f18f916663621e7f55c96f683b1870c4e693d76ba4e61e8e4942dd8688c81/share/doc/html/doc-index.json: File not found
Documentation created:
/home/colton/tmp/refined-extra/dist-newstyle/build/x86_64-linux/ghc-9.2.2/refined-with-0.3.0/doc/html/refined-with/index.html

So it succeeded but it couldn't find the doc-index.json files. I looked into the folders that it was referring to and the foo-0.0.1-guid folder (not sure what's the right term there..., I'm also assuming that the folder name has a generated guid not related to anything else) was there, but it did not have any documentation files. All of these packages, however, did have another folder with a separate guid (same version) that did have documentation files in them. The documentation folders did not contain any doc-index.json files, but everything else looked to be there. I looked at the arguments that were sent using the haddock command and they were the same folders that had the documentation in them, but since they did not have a doc-index.json file in them they showed a warning.

I tried to look into the configuration to see how the folders are selected to use in the documentation. And it looks like it pulls from the LocalBuildInfo which is made in the configuration stage (Distribution.Simple.Haddock Line 158). Whenever I looked at the creation of LocalBuildInfo I couldn't tell how they were selected other than the latest version would be chosen (Distribution.Simple.PackageIndex Line 224).

This doesn't help explain why one command would get one guid but another command would get a separate guid. So I don't know which command is wrong even though with the added warning both of them finish the documentation process successfully.

I'm also not sure if I should send the patch in to the haddock repo. That may not be what is needed there.

Any help would be appreciated

Thank you for the investigation. It would be great to fix this for 3.8, which is imminent (well, RC is). @Kleidukos: any hints? (no pressure at all :)

I have to admit that I'm absolutely lost as how it all works. I was supposed to join the Haddock team to handle the tickets, but we lost all of our knowledgeable regular contributors after I joined.

I have to admit that I'm absolutely lost as how it all works. I was supposed to join the Haddock team to handle the tickets, but we lost all of our knowledgeable regular contributors after I joined.

Hah, I fully sympathise.

@Colton-Clemmer: given that's it's supposed to be a regression in 3.6, perhaps the diff from 3.4 to 3.6 in the relevant files tells something? Unless it's huge.

TLDR: Quickjump doesn't work because the packages that it depends on were not built with the quickjump flag so it crashes when it can't find the "doc-index.json" files for the dependencies. It seems quickjump is useless without those files to point to. There's a data type in the project (InstalledPackageInfo) that defines how a package was installed and it doesn't include information about quickjump when maybe it should?

---

This has gotten kind of stale so I'll give an update on what I figured out even if I don't really have a solution. I haven't had time to mess with oss dev lately because irl stuff but I'll get back to it and probably give the other haskell repos a look, I'm just taking a few hours here and there to eventually understand how this all works, but that could take a while... (cabal as a hobby?). In the mean time I did figure out that we try to run the quickjump flag for the current project it is building but not the dependencies. It looks like quickjump needs those dependencies' "doc-index.json" files so that it can link things properly. The hex strings in the store folder are sha hashes of installed programs that correlate to a InstalledPackageInfo object that we keep in a file (I think it's .cabal/store/ghc-x.x.x/package.db/package.cache, but I'm not sure). This way multiple states of the same package and version can be installed and referenced when needed depending upon the project.

The main issue that I can find is that InstalledPackageInfo does have a property to denote whether the package was installed using haddock or not, but not whether quickjump was used. Since running quickjump for a project means that it references all of it's dependencies (the interfaces that are sent when we run the haddock command post build) and relies on the dependencies having their quickjump json files exposed it will always fail unless the project doesn't depend on anything because it needs the "doc-index.json" files for each dependency defined in the "foo.cabal" file to link everything correctly.

I'm pretty sure that there is a way to tell cabal to build everything from the ground up with quickjump. And it should have all the files that it needs after that. if we modify the InstalledPackageInfo data type to include a quickjump boolean or something similar. It would require rebuilding everything if the quickjump command was passed through but that might be the only way to get it to work. Or it could just assume that you always want quickjump and will try to build each dependency package with quickjump but not your current package unless you gave it the "--haddock-quickjump" flag. I'm not sure what the best approach is.

That said I ran into an issue trying to delete my "~/.cabal" folder so I could study how things are built up in that folder. It's giving some confusing errors but I'll just chip away at it slowly for now.

Hopefully this info will help someone that is more knowledgeable.

It sounds you are very close to a solution now. I remember we do record GHC options a package was compiled with (but only the options that are known to change the result) so that we know whether we should recompile. So perhaps we ought to record some haddock options as well and make sure they are included in the hash.

Re deleting ~/.cabal, did you also delete .ghc and any .ghc.environment.* files and dist-newstyle directories anywhere? That should be all that needs deleting.

If you want to have a smaller example, you can have a look at this repository:

• https://github.com/chshersh/ghc-plugin-non-empty

This project uses GHC 9.2.3 and, apparently, Hackage Docs builder still uses GHC 8.10 so the documentation for this project wasn't generated.

I tried to generate it manually as I always did:

cabal haddock --haddock-for-hackage

Unfortunately, I received this error:

haddock: internal error: /home/chshersh/.cabal/store/ghc-9.2.3/syb-0.7.2.1-1d2e92045eab15dcc4a155ad6d7d1fce6d39e895f6ff401ac1f29a17a9e7bd96/share/doc/html/doc-index.json: openBinaryFile: does not exist (No such file or directory)
cabal: Failed to build documentation for ghc-plugin-non-empty-0.0.0.0

Using the suggested workaround, I was able to generate the documentation tarball successfully (so I could upload it to Hackage later) with the following command:

cabal haddock --haddock-options=--quickjump --haddock-for-hackage

It's not ideal but at least it works. If fixing this issue seems to be challenging, I think in the meantime at least documenting this could be helpful 👍🏻

@gbaz: do you think we could mention the extra argument, for cabal-install 3.6 and later, at the end of https://hackage.haskell.org/upload? I don't think we need to go into which GHC version makes it necessary. I hope it doesn't break things with old GHCs.

The PR #8414 makes it possible to build a package with a working quickjump (using cabal haddock --haddock-quickjump).

There's also alternative: the cabal haddock-package command (not yet released) which allows to build haddocks with all the dependencies and a working quickjump for all packages, which is using:

• a single doc-index.json (which powers quickjump) for all local packages
• separate doc-index.json for non-local dependencies (e.g. base)

I am slightly simplifying as there are actually two flavours of the haddock-package command (one which builds a self contained directory, and one that links to hackage), but still the above is true.

Probably some time in the future it will make sense to make the cabal haddock an internal command and deprecate it and only expose cabal haddock-package (maybe it should be renamed as cabal build-docs or cabal documentation ... ) :thinking:

Given that it's fixed in master (hurray!) do we still want to help users of cabal 3.6 and 3.8? E.g., could we mention the extra argument --haddock-options=--quickjump, for cabal-install 3.6 and 3.8 (but not 3.10), at the end of https://hackage.haskell.org/upload? I don't think we need to go into an exploration which GHC version makes it necessary. I hope it doesn't break things with old GHCs, so the flag may be there always.

@gbaz, what do you think? I'm encroaching on your turf here.

Anything else to do for this ticket?

I think we should close this ticket as fixed at this point. Not too worried about instructions on the hackage webpage.

This seems to still be an issue on cabal versions 3.8.1.0 and 3.10.1.0, for which the --haddock-options=--quickjump workaround also doesn't work.

See https://github.com/haskell/haddock/issues/1582