haskell
cabal new-build does not track files added via TH’s addDependentFile
Steps to reproduce
- Download https://gist.github.com/cocreature/d07ef6f35f7bd9043db50794fe31e977.
- Build using
cabal new-build - Edit
fileinput - Try rebuilding using
cabal new-build.
Expected behavior
cabal new-build rebuilds because fileinput has changed.
Actual behavior
cabal new-build exits without rebuilding after saying Up to date.
This issue affects both cabal-2.0 as well as the master branch.
One hack solution is to watch extra-source-files too. But that won't solve dynamic (e.g. what inline-c uses?) stuff.
BTW, is there a way to force ghc --make always rerun a TH function? E.g. if I'm doing some kind of I/O at compile-time, like reading from an environment variable.
@23Skidoo that would force GHC to always build that module, that doesn't sound like a very good idea.
Yep, but sometimes you want that in a build system. For example, Shake rules can be marked alwaysRerun.
Re: extra-source-files, that field is package-global and commonly used for things like README and changelog. So we'd need to allow per-component extra-source-files to avoid unnecessary recompilation.
An alternative way to fix this is to get the list of files to be monitored from ghc --make via -ddump-module-graph (a new feature that is going to be in 8.4).
@phadej If you click that link, you'll see that I asked that exact question there.
Given that this feature will only be in GHC 8.4 I don’t think that’s a reasonable solution at this point.
@cocreature With old versions we can simulate that feature with ghc -M (which will be slower because we'll have to do dependency resolution twice - once for ghc -M and once for ghc --make, but for legacy GHC versions we can afford to not care). BTW, ghc -M does support addDependentFile dependencies:
# DO NOT DELETE: Beginning of Haskell dependencies
Data/FileEmbed.so : Data/FileEmbed.hs
Main.so : Main.hs
Main.so : Data/FileEmbed.shi
# DO NOT DELETE: End of Haskell dependencies
@23Skidoo btw, we can avoid paying for the slow-legacy fallback when it's not needed, because we know when TH is needed (we already require TemplateHaskell to be declared in *-extensions in order to enable other logic specific to TH). So I think that's a workable fallback.
Unfortunately we not only missed the boat for GHC 8.4, but most likely also for GHC 8.6 to get supporting features into GHC... as https://phabricator.haskell.org/D3898 got stuck on a decision which design path to go down... :-/
PS: Screen-scraping GHC's dump-hi output would be an obvious but terrible hack for the obvious reasons and since it's not intended to be consumed by tooling its format is neither documented nor guaranteed to be correct and can change without notice.
Upvoting this issue. It is critical for larger web-dev shops that have people who are totally focused on front-end web UI development. They require instant reload; they don't have time to poke around at files manually every time to induce rebuilds.
We are a yesod shop, so we are using stack. For comparison, stack detects this by parsing dump-hi files. The syntax change in dump-hi files in GHC 8.4 caused a similar issue in stack for a time, now fixed.
Another upvote for this issue!
I think type:bug is a more accurate description than type: enhancement. It's a regression from cabal old-build's behavior, and can cause truly wtf moments when using e.g. gitrev (see my issue #5866 ). I'd say it actually breaks the gitrev package for practical usage.
@kaldonir, no. As far as I know no-one is actively working on adding necessary features to GHC (e.g. https://github.com/ghc-proposals/ghc-proposals/pull/245 or something similar so the compiler could tell Cabal what extra dependencies exist).
Alternatively or additionally cabal could track extra-source-files, which is imprecise yet better than nothing. Again, I'm not aware anybody is working on that either.
@phadej Are you opposed to making cabal track extra-source-files as an intermediate step? This wouldn't help with gitrev but would solve this for I suspect most use cases, until we have a proper fix in GHC. I'm willing to implement this if there's support for it.
I'm not. It should be easy, I think it should go to the same place as https://github.com/haskell/cabal/commit/c2f4625f2feeda94a9174d927dc39a91c0059a7e
PR welcome!
So parsing the .dump-hi files was discouraged a while ago. I'm curious if there's any objection to just doing that, since it's what stack has been doing since 2015 with one bug report in that time.
If this is an acceptable solution then I can allocate some time and resources to implementing it.
Thanks!
GHC should be able to give cabal this info in a machine consumable cormat.
But frankly I don't really like the design of addDependentFile because it precludes proper sandboxing — it's very dynamic sort of dependency.
GHC should be able to give cabal this info in a machine consumable cormat.
Agreed, but it hasn't started doing that in the 4 years that this ticket has been open. I'm interested in solving it now.
@parsonsmatt for solving now, would simply tracking extra-source-files as suggested above suffice? That should be straightforward to implement.
Edit: hrm looks like we actually do that, but just fail to do so with wildcard matches. So I guess it suffices to patch that?
We've got ~176 files to track (and counting), and manually tracking them in the cabal file would be a major nuisance. But patching the wildcard behavior would suffice, though I do still think this should be a seamless process.
GHC's driver is being overhauled at the moment, which will help with this and other dependency issues.
Yes, I remember tracking extra source files started working again, recently. Is there a ticket about adding wildcards to extra-source-files? The closest I could find was https://github.com/haskell/cabal/issues/7290.
Yeah tracking extra-source-files is definitely the right short-term solution.
I have a pr now for glob expanding extra-src-files and am moving the remainder of the ticket to blocked on ghc, because I think we really don't want to hack around ghc proprietary formats to try to extract this information in any more sophisticated way.
Is there an easy way to instrument addDependentFile calls so we can figure out what files to add to extra-source-files as a workaround?
The type is FilePath -> Q () -- you can wrap it in any other function you want including IO actions lifted to Q to output the data however you desire.
If you're able to operate in an abstract monad, like Quasi m => m _, then you can provide a newtype around Q and define an instrumented qAddDependentFile. But this won't work for anything hardocded to Q, which is quite a lot of the API.