Automatic hoogle database for installed packages

[bos]

(Imported from Trac #402, reported by @peaker on 2008-11-11)

The hoogle package is very useful to search the standard library. Unfortunately, it does not search packages that you have installed which are not part of the stdlib.

It would be great if every package installed always had its haddock generated, and from its .txt file a .hoo file was generated (e.g: hoogle --convert), and then all .hoo files sat in some centralized directory which hoogle would then use for its searches.

This would be great documentation, and let you hoogle for anything that you can import.

[bos]

(Imported comment by @peaker on 2008-11-11)

Using hoogle --convert on the .txt files directly will not work until this bug is fixed: http://hackage.haskell.org/trac/ghc/ticket/2766

[bos]

(Imported comment by guest on 2008-11-11)

#3766 has been fixed.

More importantly, I'm not sure it was a problem. I'm still on 6.10.1, but my Hoogle had no problem converting an xmonad-contrib.txt and searching the .hoo file.

[bos]

(Imported comment by guest on 2008-12-17)

#3766 would only have been a problem for certain libraries, so it won't effect most things.

This would be great if someone could do it :-) I'll do any Hoogle changes that are necessary, but don't have time to do the Cabal side.

[bos]

(Imported comment by guest on 2008-12-17)

I don't think many Hoogle changes are necessary, but it's unclear how it's going to be handled. Is Cabal going to dump .hoo files into Hoogle's data-directory, or should Hoogle be going and looking elsewhere for .hoo files or what?

Personally, I like the idea of Cabal dumping the generated .hoo file into Cabal's data-directory myself. Simple.

But that opens other questions. As ndm says, 'Hoogle could look in the same directory as haddock does for its documentation databases, Hoogle could dump its path to stdout, hoogle could dump the converted files to its path'.

Here again I favor the simple solution of dumping the path to stdout, but another possibility is exposing Hoogle's library and thus Paths_hoogle (I think that'd work).

Finally, there becomes the question of what .hoos Hoogle should look in by default. ndm: "There should certainly be an all database, and an all libraries on hackage database, and an all installed libraries database. Which one of those is default is a good question, I'd guess at all installed libraries - but not certain."

[bos]

(Imported comment by dschoepe on 2008-12-17)

This patch should resolve this issue at least on cabal-install's part: http://article.gmane.org/gmane.comp.lang.haskell.cabal.devel/6566

The question of what is a good default for hoogle still remains though.

[bos]

(Imported comment by dschoepe on 2010-06-20)

I've been using my patch for the past few months now and it appears to work fine, so once the required haddock-patch(about generating hoogle and html output in one call[1]) is applied, this patch should be ready to be applied.

[1] http://thread.gmane.org/gmane.comp.lang.haskell.cabal.devel/5617

[bos]

(Imported comment by @dcoutts on 2010-11-06)

I applied the other patch about generating hoogle and html output in one go. I've looked at this patch but I think it needs a bit more thought.

So, the general idea is to create a binary .hoo file for each package right? Generating a merged binary database would be a separate task, not covered by this patch. There are two issues:

1. The code in the doc step of cabal-install is assuming a certain layout of the build tree by reading and writing files directly in the build tree. This will not work for packages using build-type custom. It would be ok if we were copying into a temporary image dir before doing the real installation. In that case writing extra files into the temporary image dir would be fine. As it is, I think this feature of generating a per-package .hoo db would be better in the build system itself, in the Cabal lib.
2. according to the [hoogle manual](http://www.haskell.org/haskellwiki/Hoogle#Database_Creation), the .hoo files of dependencies need to be specified as well.

Since each .txt can be converted into a .hoo, and the final .hoo is not really useful except as something to use to merge into a db that users will actually use, perhaps it would be better to just generate the .txt files per-package and then generate a big merged one. Alternatively, why generate the .txt files if we could just generate the .hoo files?

Is there any point in generating both .txt and .hoo files for each package?

More minor comments about the patch. It should use the existing framework for running programs. This would cut down on the code and integrate it properly with the Cabal exception handling and logging stuff.

[chrisdone]

@23Skidoo

So after surveying the #haskell channel it's clear that we as a community are generally completely bad at searching and getting documentation of our packages. We use hoogle online if it's available, or we google the module name if not, and locally we read docs if we have it generated, or we grep. Really, what we should be able to do is install a package and immediately be able to run hoogle ThatPackage.foo and get (1) the type information, (2) the documentation. This can be used from within GHCi. Additional to that, a hoogle search for all installed packages will trivially allow “automatically import names that I use”, among other things. It's kind of a Big Deal, it seems generally agreed, speaking in terms of productivity, to get this sorted out. So here we go.

@dcoutts

In that case writing extra files into the temporary image dir would be fine. As it is, I think this feature of generating a per-package .hoo db would be better in the build system itself, in the Cabal lib.

We have the ability to generate Hoogle from here already, so do you mean “let's change it from generating .txt files to .hoo files”? If so, that sounds good, I can do that. I already had a read through the codebase of how Haddock is implemented. I was considering implementing another command, hoogle, in the Simple setup, but given that hoogle is dependent upon Haddock (or we implement our own .txt outputter, which I think nobody wants nor would see reason to do) that it's probably reasonable to leave the hoogle generation as a flag of the Haddock command, but that we, in there, run an extra step to perform the conversion from .txt to .hoo, replacing or complementing the .txt file(s). I've seen how Cabal deals with programs, so I think I would add hoogle as a program like that, but ultimately it's a rather simple program with little configuration so it should be a few lines.

That would be step one of this issue.

Step two would be to make cabal-install, much like it does with Haddock, regenerate a database (like Haddock with its index regeneration), or merge, whatever gives us an updated hoogle database in a location specified in a similar way to how the global documentation index file's path is specified (in the config). Also there should be the config option to make this automatic a la (Documentation: True). So ideally the result would give us something like:

foo$ runhaskell Setup.hs haddock --hoogle
…
Generating documentation in: dist/html/foo/foo.html
Generating hoogle database in: dist/hoogle/foo.hoo

And then, due to Hoogle: True (or something like that), this would also work:

$ cabal install foo
Generating documentation in: dist/html/foo/foo.html
Generating hoogle database in: dist/hoogle/foo.hoo
Installing to ~/.cabal/blah/blah…
Re-generating documentation … in ~/.cabal/share/doc/index.html
Re-generating hoogle database … in ~/.cabal/share/hoogle/index.hoo
$ hoogle Foo.bar
Lorem ipsum hadoctum documentas.
Foo.bar :: Mu -> Zot

This should also, if we do it by the book according to Cabal, Just Work™ with cabal-dev and those sandboxing libraries, until Cabal has its own sandboxing released. It is important that documentation and type information reflects what I've installed, locally in a directory, or globally.

Any problems with any of the above?

[nkpart]

This sounds awesome. Is the work described here still applicable?

[m0davis]

I don't think cabal has been updated yet with the aforementioned feature. Until it is, this script may be useful: https://groups.google.com/forum/#!topic/haskell-cafe/K9wba99z_fE

[gilligan]

Will this ever be addressed again in some way ?

@m0davis : what is hoogle-install which is used in the script you mentioned. I also wonder how this could be adapted for use with sandboxes.

[treyharris]

@m0davis Your link is dead, can you update it?

[m0davis]

@gilligan sorry, I didn't notice your comment three years ago and I don't now remember much about this. @treyharris I tried and I failed. I don't know much more than you about this, but maybe you'll have better luck finding it than me. Evidently, it exists somewhere in the still-available haskell-cafe archives prior to May 2015. Perhaps also (and according to @gilligan, above), it contains the text "hoogle-install".

[kokobd]

Any update on this?

[Mikolaj]

I don't know of anybody working on this, but perhaps hoogle maintainers know of anybody.

[kokobd]

Here is a recent workaround I wrote: https://github.com/kokobd/cabal-hoogle

[Mikolaj]

Nice! What's the situation with this ticket? Is design clear enough? What is missing in your workaround to be a proper implementation?

[kokobd]

Nice! What's the situation with this ticket? Is design clear enough? What is missing in your workaround to be a proper implementation?

Not sure what the original author of this issue means, but it has been 10 years, and things have changed a lot. I believe the proper solution would be bringing a subcommand cabal hoogle to cabal projects, like stack hoogle.

There are two problems with my workaround:

1. cabal-install is not a library, so my workaround has to discover the structure of dist-newstyle on its own.
2. --enable-documentation --haddock-hoogle doesn't rebuild existing dependencies with hoogle, until I add these configs directly into ~/.cabal/config

[Mikolaj]

I have no opinion whether cabal hoogle or cabal-hoogle is better. Re 1, cabal-install now exposes a library, if that helps. Re 2, it sounds like a bug that needs fixing on its own; I wonder if there's any related ticket already.

[kokobd]

Re 1, cabal-install now exposes a library, if that helps.

Oh, it is a library in master, just not published on hackage.

I have no opinion whether cabal hoogle or cabal-hoogle is better.

Then it might be okay to close this issue and redirect users to cabal-hoogle, if the solution is found acceptable.

[Mikolaj]

Oh, it is a library in master, just not published on hackage.

3.8 is going to be released very shortly and will include it.

Then it might be okay to close this issue and redirect users to cabal-hoogle, if the solution is found acceptable.

From a cabal maintainer's perspective an external tools has many benefits, in particular for initial adoption, where the feedback loop has to be tight, but also due to separation of concerns, etc. However, I'd like to make sure cabal-hoogle gets as much help from cabal devs as an integrated command would have (which doesn't mean much, since we are understaffed:) and can influence cabal positively, so let's keep in touch, regardless if this particular ticket is closed on not.

[kokobd]

2. --enable-documentation --haddock-hoogle doesn't rebuild existing dependencies with hoogle, until I add these configs directly into ~/.cabal/config

Maybe related to #8104