Missing documentation on nimdoc.css

[FedericoCeratto]

1. nim doc fails to run when nimdoc.css is not found. The error message does not indicate how to configure the correct path
2. There is no mention of the path for nimdoc.css in nimdoc.cfg
3. ...or in the manual
4. Removing $nimdoccss from the HTML template in nimdoc.cfg does not fixes nim doc
5. compiler/docgen.nim sets the path to d.conf.getPrefixDir() / "doc" / "nimdoc.css" apparently without a way to override it

[timotheecour]

• please always try on latest devel when reporting issues and mention the corresponding hash; it saves time for whoever's looking at bug reports; also, a sample command would help for a reproducible bug report
• IIRC some of these issues should've already been fixed in devel, but hard to tell without a git hash
• https://github.com/nim-lang/Nim/pull/14324 should fix at least some of the issues you're mentioning; please try against that and let me know if you still have issues

[kaushalmodi]

The issue can be reproduced if the doc/ directory is missing in the nim installation:

oserr.nim(94)            raiseOSError
Error: unhandled exception: No such file or directory
Additional info: "/home/kmodi/nim/1.2.0/doc/nimdoc.css" [OSError]

But the error is also now more informative.

This issue is probably a duplicate of https://github.com/nim-lang/Nim/issues/13191.

[kaushalmodi]

@timotheecour This makes me think.. as the nim doc relies on the existence of that nimdoc.css, is the doc/ directory the best place for that? Should it rather live along with other sources, may be in lib/ or compiler/ ?

[timotheecour]

how was /home/kmodi/nim/1.2.0 installed? with choosenim I have: /Users/timothee/.choosenim/toolchains//nim-1.2.0/doc/nimdoc.css

is the doc/ directory the best place for that? Should it rather live along with other sources, may be in lib/ or compiler/ ?

one could make a case for either doc or lib but it probably shouldn't matter much so long nim installation installs the correct files; note that I'm also relying on dochack/dochack.{nim or js} existing for https://github.com/nim-lang/Nim/pull/14324

[kaushalmodi]

how was /home/kmodi/nim/1.2.0 installed?

I admit that it was using my local script that creates a minimal installation footprint of nim for my work.

Just for nim doc to work, I need this in my script:

cp -fP "${nim_download_dir}"/doc/*.css ./doc/. # Required for 'nim doc ..' to work

It feels rather odd that something that nim binary relies on, lives in the usually discardable (for other tools) doc/ dir.

[kaushalmodi]

note that I'm also relying on dochack/dochack.{nim or js} existing for #14324

I am looking forward to see the outcome of that PR. Right now, I have a custom config.nims docs task that Does The Right Thing :)

[timotheecour]

I'm fine with the idea of a miminal footprint install, but this really should be done via some API (tested in testament), not by guessing which files to include, otherwise things are guaranteed to break at some point (like here); nim should have some freedom to keep internal details as implementation details, while providing an API to do things like that.

That's the more robust thing IMO (moving around nimdoc.css would not address future internal changes, including that new dochacks.nim/js dependency)

I am looking forward to see the outcome of that PR

the idea is for index+search to "just work" for any project with a single command (and making --index on be new default soon after)

have a custom config.nims docs task that Does The Right Thing :)

hopefully https://github.com/nim-lang/Nim/pull/14324 will simplify your script and make it more robust :) (dochack.nim should be tied to your nim repo, not to a fixed origin like "https://nim-lang.github.io/Nim/dochack.js eg to keep dochack.nim consistent with your local changes; also #14324 handles --project

one thing I've left as TODO is fixing href's to dochack.js in case of nested dirs; help welcome there

closing this issue

So what's needed to close this PR, assuming that OP was using non standard installation if I'm reading https://github.com/nim-lang/Nim/issues/14424#issuecomment-632700091 correctly? we already have ./koch zip, do we need ./koch zip --minimal ?

[kaushalmodi]

I'll let the OP comment on if this issue can be closed. For me, personally, I understood the risks, and adapted the minimal download script for my own use. I don't think that the koch should be cluttered with that switch.

[kaushalmodi]

hopefully #14324 will simplify your script and make it more robust :) (dochack.nim should be tied to your nim repo, not to a fixed origin like "https://nim-lang.github.io/Nim/dochack.js eg to keep dochack.nim consistent with your local changes; also #14324 handles --project

+1

I will follow the development of that PR. I looks like a much involving PR, so I probably cannot help much there. But I can help test out the PR. We can continue discussion about that PR in its thread.

[FedericoCeratto]

@timotheecour the issue was found in this nightly build: https://github.com/nim-lang/nightlies/releases/tag/2020-05-20-version-1-2-7800fa3 which is designate as rc1 for 1.2.2

[FedericoCeratto]

Note to self: https://nim-lang.github.io/Nim/packaging.html should be updated with the solution

[shirleyquirk]

compiler expects nimdoc.css to be in the same tree as system.nim

https://github.com/nim-lang/Nim/blob/5d95137ce040edd181e56bc132ce319e59bd1ee4/compiler/nimpaths.nim#L23-L24 i.e. /usr/lib/nim/doc

but no mention of this in docs and absent instructions to the contrary e.g. Arch puts them under /usr/share/nim/doc

https://forum.nim-lang.org/t/7679

[timotheecour]

@shirleyquirk please cross-link to avoid splitting discussion; i did it for you here: https://forum.nim-lang.org/t/7679#48913

choosenim installs it to: /Users/timothee/.choosenim/toolchains//nim-1.4.4/doc/nimdoc.css

Arch puts them under /usr/share/nim/doc

maybe that's the bug?

can you show output of

find /usr/share/nim/ ?

[shirleyquirk]

Debian (sid,unstable, 1.4.2) puts it in /usr/lib/nim/doc, which probably doesn't work, either. arch provides a symlink meaning /usr/lib/nim/doc/nimdoc.css exists, referencing a bug from version 0.13 and the 'nim doc2' command.

installation instructions say "The expected stdlib location is /usr/lib/nim" which is sensible.
I know @timotheecour is familiar with this as i see you've worked on this problem already! just familiarizing myself, and not remotely aware of all the ramifications but: i see docCss is hardcoded as $nimr/doc/nimdoc.css $nimr is calculated based on the location of the nim binary: https://github.com/nim-lang/Nim/blob/64e6670de45f8f5a470e5cc3ad9ed2cb85f0b57c/compiler/options.nim#L627-L640 so if the compiler binary is in /usr/bin/nim then the compiler expects to find nimdoc.css under /usr/doc/nimdoc.css and if /usr/local/bin/nim then /usr/local/doc/nimdoc.css

it would be antisocial to put nims docs under /usr/doc meaning the whole nim tree would have to go under /opt with the other antisocial packages, but that would require changes to all distro's installations

the path of least resistance is probably to add a check here in docgen.nim to look under nimr/lib/nim as well if file not found?

[timotheecour]

can't we fix the distro's installation scripts so that they don't shuffle the installation files in a platform specific way? there are other things besides nimdoc.css that need to be in a specific location (eg dochack.js, in the future more things as well eg if we want to use karax for nim doc, etc)

in ideal world, every installed nim (via choosenim or otherwise) respects the original layout

[shirleyquirk]

yes, that's possible, change the packaging instructions to put the whole nim tree in /opt or somewhere. i dont like that solution, it multiplies the amount of work by the number of distros, would probably break a lot of users installations, and goes against conventional expectations (binary under /usr/bin, library under /lib, headers under /usr/include) it's understandable that a packager would expect 'doc' to contain the documentation, indeed it's listed as optional in the packaging info, but if nimdoc.css and dochack etc are requirements for the toolchain, they should live under /lib/nim imho.

[timotheecour]

can you point to the source code where those packaging instructions are written?

[shirleyquirk]

https://github.com/nim-lang/Nim/blob/devel/doc/packaging.rst

[kaushalmodi]

@shirleyquirk

they should live under /lib/nim imho

I agree! As I mentioned earlier, to anyone unknown with this special dependency, the doc directory would look like something optional and not a requirement for Nim commands to work.

@timotheecour Can the doc related files like css and js be moved to a doc/ subdir in lib?

[timotheecour]

https://github.com/nim-lang/Nim/blob/devel/doc/packaging.rst

these instructions are out of date, and should be updated (eg to use sh build_all.sh among other things) we in fact shouldn't rely on bash commands to run by hand, this should be abstracted out by ./koch command instead of things like this:

for fn in nimble nimsuggest nimgrep; do cp ./bin/$fn <tempdir>/nim/bin/; done

they should live under /lib/nim imho

where would like like nimdoc.css to be inside nim repo? ($nim/lib/nim does not exist)

there are other dependencies besides $nim/lib eg $nim/tools/dochack/dochack.nim

again, ideally files are copied as is so that relative paths wrt nim binary work regardless of the installation platform; that said, it would be ok to reorganize a bit some of files nim relies on so that they're all inside some subdirectory of $nim/ (or installation instructions could simply copy the whole tree $nim/ for simpliicty)

[kaushalmodi]

($nim/lib/nim does not exist)

I actually meant $nim/lib/doc/. Can that hierarchy be created if it doesn't exist?

[timotheecour]

I'd be ok with $nim/lib/extras/ or $nim/extras/ as it shouldn't be restricted to doc stuff there are 2 option:

option 1: extras=$nim/lib/extras/ option 2: extras=$nim/extras/

with either option we can write a PR as follows:

cd $nim
mkdir $extras
mv tools/dochack $extras
mv doc/nimdoc.css $extras
mv doc/{advopt.txt,basicopt.txt} $extras # technically not needed but cleaner
update references

anything else missing?

note that option 1 vs 2 matters because of the nim --lib:pathtolib flag, ie, with option 1, you'd expect paths under $extras to honor --lib, with option 2, those paths would honor nim binary location, not --lib

[kaushalmodi]

@timotheecour

option 1 vs 2 matters because of the nim --lib:pathtolib flag

This makes it less appealing to ship the .css file in the lib/ dir. I would expect nim doc .. command to Just Work whether or not the user uses --lib:<..>.

So this leads me to https://tldp.org/HOWTO/HighQuality-Apps-HOWTO/fhs.html .. Would it then be better to put things in /etc/nim/? It essentially what you suggested : extras/ but with a more standard name for Linux distributions.

etc/: Contains configuration files. If your Software uses several files, put them under a subfolder like /etc/myproduct/

[timotheecour]

This makes it less appealing to ship the .css file in the lib/ dir. I would expect nim doc .. command to Just Work whether or not the user uses --lib:<..>.

that's not what I'm saying; --lib is an optional flag controlling location of lib folder which can override the one inferred otherwise from nim binary:

nim doc main # lib = $nimbinary.parendDir.parendDir/"lib"
nim doc --lib:pathtolib main # lib = pathtolib 

it boils down to whether extras are tied to nim binary or to stdlib distribution

[cvanelteren]

I am running arch, and I am new to nim. Today, I ran into the issue when attempting to generate docs, how can I fix this?

[timotheecour]

how can I fix this?

PR welcome to proceed as described here: https://github.com/nim-lang/Nim/issues/14424#issuecomment-849803994

[shirleyquirk]

$nim is /usr if the nim binary is /usr/bin/nim and it's the root directory (!) if the binary is in /bin/nim it's not appropriate to add folders or files either of those places.

/usr/lib/nim/ or /etc/nim are reasonable, but neither of the options proposed by @timotheecour.

cd $nim/lib
mkdir nim
mkdir nim/extras
# or mkdir nim_extras ?
...

Is what I'm proposing, would that be acceptable?

[timotheecour]

$nim is /usr if the nim binary is /usr/bin/nim and it's the root directory (!) if the binary is in /bin/nim it's not appropriate to add folders or files either of those places.

see how setDefaultLibpath is implemented, it infers $lib from nim binary but also takes into account a few hard coded locations so that it does the right thing eg: if nim is in /usr/bin/nim, $lib will be /usr/lib/nim

• right now it doesn't handle /bin/nim but this could be done if support for this is needed

• the problem is that only $lib is remapped correctly, not $nim, so anything under $lib would result in relative paths resolving correctly between clone and installation, but for paths under $nim this currently doesn't work.

• $nim/lib/extras/ would therefore be an easy solution, but one problem is that this won't have desirable semantics when user passes --lib flag, as files like nimdoc.css should be tied to compiler binary, not to the standard library

We could apply a similar logic to getPrefixDir as the one done for setDefaultLibpath (factoring out common code), that way, $nim resolves correctly regardless of whether it's from a clone or from an installation, but that would break relative paths of the form ../ from a path in lib to a path in extras.

Given that, why can't we just copy all whitelisted files during installation by retaining the original layout?

example 1:

nim binary: /usr/bin/nim

$nim becomes: /usr/lib/nim and contains everything that's whitelisted, without messing with original layout: /usr/lib/nim/doc/advopt.txt /usr/lib/nim/lib/system.nim /usr/lib/nim/nimbase.h /usr/lib/nim/doc/nimdoc.css

example 2:

nim binary: /usr/local/bin/nim ditto, you'd have /usr/local/lib/nim/doc/nimdoc.css etc

this would work regardless of whether extras goes under $nim/extras or $nim/lib/extras (and extras wouldn't even be needed anymore), and relative paths (including with ..) would keep working.

[FedericoCeratto]

Workaround: Locate the files in the Nim sources and copy them to:

• /usr/doc/nimdoc.css
• /usr/tools/dochack/dochack.nim
• /usr/tools/dochack/fuzzysearch.nim

Then compile dochack in the same directory nim js -d:release /usr/tools/dochack/dochack.nim

[FedericoCeratto]

nim doc is used to generate documentation for both the compiler itself but also 3rd party libraries and tool. People should be able to supply custom nimdoc.css files for each project by overriding the default file path.

Regardless, the default paths in the Nim codebase should allow overriding by package maintainers. Nim should not hardcode paths in undocumented consts or make assumptions or guesswork. The locations should be documented in https://github.com/nim-lang/Nim/blob/devel/doc/packaging.rst.

Can we gather all the "const" paths in a single installation_paths.nim file for easy patching (at packaging time)?