make docbuilder work with old index as fallback (& add admin acct to trustees)

[mitchellwrosen]

Hi, I'm experimenting with a private Hackage server at the moment, and I'm struggling to get doc building working.

I tried running hackage-build locally as rundocs.sh does, but I see this output:

There are 1 packages with a total of 1 package versions
So far we have built or attempted to built 0 packages; only 1 left!
Considering the most recent version only:
   0 built succesfully
   0 failed to build  
   1 not yet built    
Considering all versions:
   0 all versions built successfully                       
   0 attempted to build all versions, but some failed      
   0 not all versions built yet, but those that did were ok
   0 not all versions built yet, and some failures         
   1 no versions built yet                                 
0 package(s) to build

I dug a little bit deeper, and it appears hackage-build is grabbing package indexes from ~/.cabal/config/packages here:

https://github.com/haskell/hackage-server/blob/af9096e00bf251c6919e568b0d12441e6715cb65/exes/BuildClient.hs#L160

So, this explains why I got 0 package(s) to build - the package uploaded to my private hackage is called kab, which does not exist on hackage.haskell.org.

Any advice on how to proceed here is appreciated!

[gbaz]

If you add your local cabal server to the ~/.cabal/config file as a remote repo, then it should have its packages pulled in as well, which I think should suffice?

If you find this works, a PR to improve the documentation here is very welcome :-)

[mitchellwrosen]

Here's the latest:

• I added

  repository whobar
    url: http://localhost:8080/
  

  to /.cabal/config and re-ran hackage-build build. Same problem ("0 packages to build").
• Ok, I suppose I should cabal new-update first to pull the index into ~/.cabal/packages
• Now I have a folder ~/.cabal/packages/whobar
• hackage-build build, same error
• Oh, it looks for a 01-index.tar, but I only have an 00-index.tar...
• Looks like I have to use TUF. I generated some keys, threw them in the datafiles dir and updated my repository config to:

  repository whobar
    url: http://localhost:8080/
    secure: True
    root-keys:
    key-threshold: 0
  

• hackage-build build... 1 package to build!
• Fails with:

  Building kab-0
  Build results for kab-0:
  /root/hackage-server/build-cache/results/kab-0.report
  no docs
  /root/hackage-server/build-cache/results/kab-0.log
  hackage-build: (4,0,3) Forbidden http://localhost:8080/package/kab-0/reports/
  Error: Forbidden
  
  No access for this resource.
  

(Btw, the haddock build is supposed to fail here). So it is getting a 403 upon trying to upload the failed build report to the server?

As always, thanks for the help.

[gbaz]

We could tweak the source to fallback to 00-index if there's no 01-index.

Vis a vis the 403, you need to make sure the user specified in the configfile (at bo_stateDir opts </> "hackage-build-config") is in the trustee group of your hackage server.

[gbaz]

Once you get this working, a PR that gives simple instructions to the README.md (or in the docs directory, i guess, even though that's pretty stale) would be very appreciated!

[mitchellwrosen]

Whew, it's working now, after adding admin to the trustees group. Yay!

I can add some notes to the README shortly. About falling back to 00-index.tar - this would be nice, TUF feels unnecessary for a private hackage instance wherein I trust my VPN and/or ssh keys to be routing me to a trusted server.

[gbaz]

Here's where we'd need to fix to add fallback logic: https://github.com/haskell/hackage-server/commit/cf5310ab9b72fe45f6830c0a04bc6615aa146424#diff-7e7aeeccaf1336a53230bac93394a5cf

I don't remember exactly why we thought switching to the 01-index was particularly worthwhile, but I think it improved some corner cases because of that file contains more revision information.

It's also sort of silly/irritating that the default admin account isn't automatically in trustees too -- a PR to change that would also be welcome :-)

[gbaz]

i did add 00-index as a fallback in a recent pr now. but we need to remove the limitation that 00-index doesn't have everything 01 does, for back-compat reasons that i think we can now remove.

[gbaz]

we removed that limit too now, so this is just about adding admin to trustees.