[docu] Add new ROOT build option: documentation-building

[ferdymercury]

This Pull request:

Changes or fixes:

This allows to build the documentation as part of the normal ROOT build and sets the basis for integrating it into the GitHub CI. And to test new PullRequests so that they do not break the existing documentation nor do add new warnings to it.

You can still build the documentation as an external project, like is done now.

Status

Draft, still work to do on the notebook / tutorial part.

History

This supersedes the old PR https://github.com/root-project/root/pull/9966 that I just closed.

The goals were:

• [x] Migrate old Makefile to modern CMake. See https://github.com/root-project/root/issues/9090
• [x] Allow building the documentation from a git source repository with read-only permissions. I.e. try to not pollute the sources within the process. This follows the Make to CMake migration philosophy. See https://github.com/root-project/root/issues/8947
• [x] As done by ALICE, use a 'dynamic' Doxyfile declaration, that only specifies what needs to be changed with respect to the default one. This makes easier the maintenance, as you do not need to constantly update the Doxyfile when a new doxygen version arises, and prevents warnings when running in older doxygen versions. This will hopefully contribute to next point:
• [ ] https://github.com/root-project/root/issues/9953 in combination with https://github.com/ammaraskar/gcc-problem-matcher
• [ ] Allow in the future for automatic meta-documentation of the CMake flag system. See https://github.com/root-project/root/issues/8999
• [x] Potentially add a flag in the main ROOT CMakeLists.txt, to enable the building of the documentation via a normal "add_subdirectory()", so that one does not need to build as a separate process.
• [x] Make the doxygen documentation thread-safe
• [x] Don't let doxygen search for "input" files in the output directory (e.g. it looks now for Mathjax output files that one just "generated")

Checklist:

• [x] tested changes locally
• [ ] We would need to update https://github.com/root-project/web/blob/main/for_developers/doxygen/index.md
• [ ] Update rootspi scripts to build docu in CI accordingly
• [ ] tutorial generation does not work yet as "out of source". To-do...
• [x] require min doxygen version 1.9.4
• [ ] pre-check that output directory is empty ?
• [ ] if GVIZ not found, disable dot
• [ ] Stamp file generation to only recompile when changes
• [x] Remove unneeded unnecessary files Images.dat etc ? (or prevent that they appear in root source)
• [x] Make filter.cxx threadsafe but giving unique names to ImageSizes.dat .root etc
• [x] set NUM_PROC_THREADS
• [x] listLibs pass ROOT_CMD as variable
• [ ] Understand why libs.C does not find classes such as TBrowserTimer in libCore.so
• [x] check why there are stdout.dat and NumberOfImages not being removed
• [ ] Consider enabling https://www.doxygen.nl/manual/config.html#cfg_clang_assisted_parsing
• [ ] Fix strip_from_path in root-tag https://github.com/root-project/root/issues/10059
• [x] Check compress https://github.com/root-project/root/issues/10066
• [x] Upgrade mathjax https://github.com/root-project/root/issues/10067

[github-actions[bot]]

Test Results

    11 files      11 suites   2d 4h 48m 45s :stopwatch:  2 635 tests  2 635 :white_check_mark: 0 :zzz: 0 :x: 27 419 runs  27 419 :white_check_mark: 0 :zzz: 0 :x:

Results for commit a9ba8e75.

:recycle: This comment has been updated with latest results.

[couet]

@ferdymercury is this new mechanism, using cmake, ready to replace the current Makefile technique? if yes how to test it?

[ferdymercury]

@ferdymercury is this new mechanism, using cmake, ready to replace the current Makefile technique? if yes how to test it?

It's almost ready. Just missing the notebook and tutorial part.

To test it, just clone my branch.

Then:

Compile ROOT with CMake flag -Ddocu=ON -DDOCU_LOCATION=/tmp/docoutput -DDOCU_THREADS=8 -DDOCU_LOGFILE=/tmp/warnings.txt -DDOXYGEN_EXECUTABLE=/opt/doxygen/build/bin/doxygen -DDOCU_INPUT="./mainpage.md;../../core"

You probably also need to disable WebCanvas in your .rootrc

Note, only specify DOCU_INPUT if you want to build a small part of the docu, to be quicker. Otherwise, it will get the default, which is to build all folders.

[couet]

I will try when ready

[ferdymercury]

I will try when ready

I'd say it's ready to try now.

[couet]

@ferdymercury I took the branch from https://github.com/root-project/root/pull/15314 . I issued the command suggested in you previous comment:

cmake ../root-new-doc -Ddocu=ON -DDOCU_LOCATION=/tmp/docoutput -DDOCU_THREADS=8 -DDOCU_LOGFILE=/tmp/warnings.txt -DDOXYGEN_EXECUTABLE=/Users/couet/bin/doxygen -DDOCU_INPUT="./mainpage.md;../../core"

and I get:

CMake Error at documentation/doxygen/CMakeLists.txt:58 (STRING):
  STRING sub-command REGEX, mode REPLACE needs at least 6 arguments total to
  command.

[jolly-chen]

@ferdymercury This seems to be caused by gROOT->GetGitBranch\(\) failing at https://github.com/root-project/root/blob/eb2d443a5d021500f1085df78ca52701007c263c/documentation/doxygen/CMakeLists.txt#L46 because there is no root.exe yet when building doxygen as a step of the ROOT build.

@couet Can you try building without documentation first (-Ddocu=off), run source bin/thisroot.sh, then build again with -Ddocu=on)? This should work.

[ferdymercury]

@ferdymercury I took the branch from #15314 . I issued the command suggested in you previous comment:

cmake ../root-new-doc -Ddocu=ON -DDOCU_LOCATION=/tmp/docoutput -DDOCU_THREADS=8 -DDOCU_LOGFILE=/tmp/warnings.txt -DDOXYGEN_EXECUTABLE=/Users/couet/bin/doxygen -DDOCU_INPUT="./mainpage.md;../../core"

and I get:

CMake Error at documentation/doxygen/CMakeLists.txt:58 (STRING):
  STRING sub-command REGEX, mode REPLACE needs at least 6 arguments total to
  command.

Thanks for the review! This might be a platform-dependent issue, it works well in Ubuntu. Or it might be that you are using a tarball instead of git repository?

So the conflicting line is:

STRING(REGEX REPLACE "^\n\\(const char \\*\\) " "" ROOT_GIT_VERSION ${ROOT_GIT_VERSION})

which is called after:

execute_process(COMMAND ${ROOT_root_CMD} -l -b -q -e gROOT->GetGitBranch() OUTPUT_VARIABLE ROOT_GIT_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE)

Could you post here the output of ROOT_GIT_VERSION before calling REPLACE ? So that I know what to fix.

[ferdymercury]

@jolly-chen ohh I see what you mean.

Maybe it's better then to add a dependency on root.exe for this ?

[jolly-chen]

@ferdymercury Yes I think this would be fixed with this todo https://github.com/root-project/root/blob/eb2d443a5d021500f1085df78ca52701007c263c/documentation/doxygen/CMakeLists.txt#L22

[ferdymercury]

@couet Can you retry now after applying this commit ?

https://github.com/root-project/root/pull/15160/commits/a720d7969d4b0874065d41c0e398ffa9ab565e52

[couet]

I'll try and let you know

[couet]

I am a bit lost now .. this branch has conflicts ... how to get again the latest version ... ? there is this PR and this one: https://github.com/root-project/root/pull/15314 that's very confusing

[ferdymercury]

I am a bit lost now .. this branch has conflicts ... how to get again the latest version ... there is this PR and this one #15314 that's very confusing

I've resolved the conflicts now. Try with this one, the other PR is just for later on.

[couet]

So, I cloned the ROOT repo in the folder docgmake. Then in that cloned folder I did gh pr checkout 15160 to get this branch. Then cd ..; mkdir docgmake-bin; cd docgmake-bin. And in that bin folder I issued the command:

cmake ../docgmake -Ddocu=ON -DDOCU_LOCATION=~/rootdoc -DDOCU_THREADS=8 -DDOCU_LOGFILE=/tmp/warnings.txt -DDOXYGEN_EXECUTABLE=/Users/couet/bin/doxygen -DDOCU_INPUT="./mainpage.md;../../core"

then make -j8

which gives me:

...
[100%] Built target modules_idx
[100%] Generating tutorials/hsimple.root

Processing hsimple.C...
hsimple   : Real Time =   0.23 seconds Cpu Time =   0.08 seconds
(TFile *) 0x7f77c78bd300
[100%] Built target hsimple
/bin/sh: -DROOT_COMMAND=\"ROOT_root_CMD-NOTFOUND\": command not found
make[2]: *** [documentation/doxygen/CMakeFiles/Preparation] Error 127
make[1]: *** [documentation/doxygen/CMakeFiles/Preparation.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[100%] Built target PreparationPyz
make: *** [all] Error 2

So it seems the gmake command was wrong.

[jolly-chen]

@couet can you try again with this commit https://github.com/root-project/root/pull/15160/commits/da41596be8c0c73bf52de6a1d874e67b4c699dea?

(sorry accidentally closed the PR)

[couet]

I am rebuilding. By the way, why do we need to specify DOXYGEN_EXECUTABLE ? If a valid oxygen version is in $PATH we should not be obliged to specify this variable.

[ferdymercury]

@couet I just added two more commits, let me know if they fix the issue you were seeing. Thanks!

[ferdymercury]

By the way, why do we need to specify DOXYGEN_EXECUTABLE ? If a valid oxygen version is in $PATH we should not be obliged to specify this variable.

Thanks, good idea. I just added a commit to support that.

[couet]

and:

DOCU_LOCATION DOCU_LOGFILE

do they have default values ? DOCU_LOGFILE dinot exist before but DOCU_LOCATION did exit and the default value was ~/rootdoc

[ferdymercury]

DOCU_LOCATION

Should be fixed now. Not specifying anything defaults to ~/rootdoc

DOCU_LOGFILE dinot exist

If you do not specify it, warnings will be printed to terminal. If specified, warnings will be printed to a file on your disk.

[couet]

Now I get:

...
[100%] Built target modules_idx
[100%] Generating tutorials/hsimple.root

Processing hsimple.C...
hsimple   : Real Time =   0.22 seconds Cpu Time =   0.07 seconds
(TFile *) 0x7fccdab5e360
[100%] Built target hsimple
filter.cxx:74:10: fatal error: 'unistd.h' file not found
#include <unistd.h>
         ^~~~~~~~~~
[100%] Built target PreparationPyz
1 error generated.
make[2]: *** [documentation/doxygen/CMakeFiles/Preparation] Error 1
make[1]: *** [documentation/doxygen/CMakeFiles/Preparation.dir/all] Error 2
make: *** [all] Error 2

I am on MacOS

[ferdymercury]

'unistd.h' file not found

Ok, we might need to add some platform-dependency now.

When you run "cmake", what is the CXX compiler that is reported in the terminal?

Or in other words, what is the value of CMAKE_CXX_COMPILER in your case? In my case it's c++

Might be related to https://stackoverflow.com/questions/29357618/clang-fatal-error-unistd-h-file-not-found

[ferdymercury]

[100%] Built target hsimple filter.cxx:74:10: fatal error: 'unistd.h' file not found #include <unistd.h> ^~~~~~~~~~ [100%] Built target PreparationPyz 1 error generated.

@couet I tried building with mac12 and I cannot reproduce the error you see. See the CI log: https://github.com/root-project/root/actions/runs/8877537165/job/24371327147?pr=15314 The documentation is built correctly (aside from the fact that 'graphviz-dot' is missing).

Could it be something related to mac14 vs mac12 ?