Parallelize Haddock builds across modules?

[bgamari]

Facebook has a rather intriguing commit in their Haddock repository which appears to allow parallelization of the Haddock build. It would be worth checking whether this can be folded upstream as Haddock builds tend to be a significant cost center during larger builds.

[alexbiehl]

I want to give this a try. In order to clean this up maybe we can clarify the relationship between Hsc and Ghc monad? Would be good to be able to safely lift one into the other to nicely fit the hooks and processModules types. Tagging @hsyl20 as he seems be working that area in Ghc.

[hsyl20]

I don't understand why we need both Ghc/Hsc monads with Session/HscEnv states. It's probably an historical artifact from when hsc was a separate program from the ghc driver. We should try to replace them with a single pure state monad and avoid the use of IORef.

[hsyl20]

Actually Ghc is a state monad but Hsc is only a Reader monad so we should probably keep them separate.

[alexbiehl-gc]

we should probably keep them separate

Does it make sense to have a function runGHC :: Ghc a -> Hsc a or would it violate invariants with Hsc?

[hsyl20]

In general no. We can safely lift a Hsc a into a Ghc a but not the other way around. EDIT: we can if we drop the environment resulting from the execution of Ghc a.

Btw I don't think this patch works with current GHC head because hscFrontendHook is unused since https://gitlab.haskell.org/ghc/ghc/-/commit/eb629fab96fbff43f79190767731501d8642f524

[alexbiehl-gc]

Btw I don't think this patch works with current GHC head because hscFrontendHook is unused since

Good catch. Seems like we either want to reinstate that or add something that fits better for our use case first. Open for ideas and happy to take it on.

[hsyl20]

Looking at parseModule and typecheckModule, both could be in Hsc monad instead of Ghc monad. Only loadModule needs to modify the HscEnv (to add the module in the home package table). So processModule could be in Hsc too. What I would suggest:

1. fix hscFrontendHook :)
2. add and expose hscParseModule and hscTypecheckModule. Use them in parseModule and typecheckModule: basically parseModule args = getSession >>= \hsc_env -> runHsc hsc_env (hscParseModule args)
3. fix the patch to use hscParseModule/hscTypecheckModule and let processModule be in Hsc

[hsyl20]

It could be useful to define liftHsc :: GhcMonad m => Hsc a -> m a.

[pepeiborra]

Btw I don't think this patch works with current GHC head because hscFrontendHook is unused since

Good catch. Seems like we either want to reinstate that or add something that fits better for our use case first. Open for ideas and happy to take it on.

@watashi suggested reimplementing this patch as a sourcecode/typechecker plugin, but we haven't explored the idea since hooks worked just fine.

[alexbiehl]

@watashi suggested reimplementing this patch as a sourcecode/typechecker plugin

This is a great idea. I have checked up on Plugin and it offers

typeCheckResultAction :: [CommandLineOption] -> ModSummary -> TcGblEnv -> TcM TcGblEnv

which seems to have everything we need to generate the Haddock interface. I am going to try this approach during holidays.

[watashi]

For reference, I had an old local commit created to use plugins:

https://github.com/watashi/haddock/commit/febba04aa046b1192e4a2013919bd7131b769539

It's not completed, at minimum, it missed the Opt_PluginTrustworthy (https://github.com/ghc/ghc/commit/406e43af2f12756c80d583b86326f760f2f584cc) flag to plugins to get the correct SafeHaskellMode for a module.

[duog]

Note hscFrontendHook will be back in 9.2, see https://gitlab.haskell.org/ghc/ghc/-/commit/adaa6194753f33a705ac57cd8ddb94dc9aff1f54

[alexbiehl-gc]

Over the holidays I made haddock a GHC type checker plugin, very loosly based on @watashis work. It's looking good but I found some weirdness around handling of declarations without type signatures as well as bundled pattern synonyms. You can find the work here https://github.com/haskell/haddock/tree/wip/alex/plugin I will continue to iron out the quirks and land this eventually.

Making it a type checker plugin instead of a frontend plugin also saves some work on Haddocks side which is nice.

[watashi]

@alexbiehl-gc Do you have any update or your patch or are you blocked by any issue there?