RIOT-OS
dist/tools: use `doc-ci` in the doccheck if possible
Contribution description
Pull Request #21300 introduced Doxygen as a package, which can be used with the make doc-ci command. In order to avoid static test fails, the doccheck script should use doc-ci as well, otherwise errors will be generated in the static tests that would not be an issue in the CI.
I extended the version warning message and gave a hint for the user that make doc-ci can be used to build the Doxygen documentation with the latest Doxygen version:
cbuec@W11nMate:~/RIOTstuff/riot-doxygen/RIOT$ make -C doc/doxygen doxygen-version
make: Entering directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
Warning: Doxygen version 1.12.0 is too old. It is recommended to use at least version 1.13.2 to avoid incorrectly formatted output.
You can use 'make doc-ci' to build the documentation with the required Doxygen version, which will be downloaded and (if required) compiled locally.
make: Leaving directory '/home/cbuec/RIOTstuff/riot-doxygen/RIOT/doc/doxygen'
Testing procedure
Run make static-test and observe that it downloaded Doxygen:
cbuec@W11nMate:~/RIOTstuff/riot-doxygen/RIOT$ ls -l dist/tools/doxygen/
total 286352
-rw-r--r-- 1 cbuec cbuec 3726 Apr 7 13:33 Makefile
-rwxr-xr-x 1 cbuec cbuec 215163480 Apr 7 14:10 doxygen
drwxr-xr-x 6 cbuec cbuec 4096 Jan 9 20:16 doxygen-1.13.2
-rw-r--r-- 1 cbuec cbuec 78038269 Jan 9 20:41 doxygen-1.13.2.linux.bin.tar.gz
Also the doccheck will complain about obsolete parameters, since the Doxyfile has not been updated yet (see #21292):
cbuec@W11nMate:~/RIOTstuff/riot-doxygen/RIOT$ ./dist/tools/doccheck/check.sh
ERROR: Doxygen generates the following warnings:
warning: Tag 'OUTPUT_TEXT_DIRECTION' at line 102 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'HTML_TIMESTAMP' at line 1366 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'FORMULA_TRANSPARENT' at line 1671 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'LATEX_SOURCE_CODE' at line 1984 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'LATEX_TIMESTAMP' at line 2000 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'RTF_SOURCE_CODE' at line 2074 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOCBOOK_PROGRAMLISTING' at line 2179 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'CLASS_DIAGRAMS' at line 2387 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOT_FONTNAME' at line 2429 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOT_FONTSIZE' at line 2436 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOT_TRANSPARENT' at line 2684 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
sys/include/net/gnrc/netif/pktq.h:14: warning: Potential recursion while resolving \ref command!
cpu/rpx0xx/include/periph_cpu.h:234: warning: unable to resolve reference to 'gpio_t' for \ref command
...
Issues/PRs references
Required for #21292.
Murdock results
:heavy_check_mark: PASSED
8446315bc04f0caa33fa80cb0454ecbec1b89950 dist/tools/doxygen: use dlcache for Doxygen download
| Success | Failures | Total | Runtime |
|---|---|---|---|
| 1 | 0 | 1 | 01m:22s |
Artifacts
In order to avoid static test fails, the
doccheckscript should usedoc-cias well, otherwise errors will be generated in the static tests that would not be an issue in the CI.
So why not using doc-ci always instead of only when doxygen-version returns non-zero?
In order to avoid static test fails, the
doccheckscript should usedoc-cias well, otherwise errors will be generated in the static tests that would not be an issue in the CI.So why not using
doc-cialways instead of only whendoxygen-versionreturns non-zero?
When doxygen-version returns no warning, the local Doxygen version is already up to date and there would be no need to download the 78M binary archive from Github.
But I just saw that there should be an else that covers just that 😅
When doxygen-version returns no warning, the local Doxygen version is already up to date and there would be no need to download the 78M binary archive from Github.
True, but only for the first run and I believe there would be value in having make static-test match the output of the CI, even at the cost of 78 MiB download on the first run (until wiping the cache or bumping the min release).
Okay, I'm convinced :D
But I would still like to keep the changes to the doc/doxygen/Makefile. Perhaps in a separate commit now?
But I would still like to keep the changes to the
doc/doxygen/Makefile. Perhaps in a separate commit now?
Makes sense to split this out I'd say.
Urgh, this seems to need a bit more attention. Apparently Doxygen 1.13.2 has a bug that doesn't parse code blocks correctly:
I'll have to try it with latest, but not tonight.
For example from https://github.com/RIOT-OS/RIOT/blob/master/sys/psa_crypto/doc.md?plain=1:
sys/psa_crypto/doc.md:463: warning: unable to resolve reference to 'porting-software' for \ref command
sys/psa_crypto/doc.md:464: warning: unable to resolve reference to 'porting-hardware' for \ref command
sys/psa_crypto/doc.md:465: warning: unable to resolve reference to 'porting-secure-elements' for \ref command
sys/psa_crypto/doc.md:560: warning: Unsupported xml/html tag <libraryname> found
sys/psa_crypto/doc.md:563: warning: Unsupported xml/html tag <libraryname> found
sys/psa_crypto/doc.md:589: warning: explicit link request to 'if' could not be resolved
sys/psa_crypto/doc.md:590: warning: explicit link request to 'include' could not be resolved
sys/psa_crypto/doc.md:591: warning: explicit link request to 'endif' could not be resolved
sys/psa_crypto/doc.md:597: warning: explicit link request to 'if' could not be resolved
sys/psa_crypto/doc.md:598: warning: explicit link request to 'include' could not be resolved
sys/psa_crypto/doc.md:600: warning: Unsupported xml/html tag <library_context_type_t> found
sys/psa_crypto/doc.md:601: warning: explicit link request to 'endif' could not be resolved
sys/psa_crypto/doc.md:906: warning: explicit link request to 'if' could not be resolved
sys/psa_crypto/doc.md:908: warning: explicit link request to 'endif' could not be resolved
sys/psa_crypto/doc.md:909: warning: explicit link request to 'if' could not be resolved
sys/psa_crypto/doc.md:911: warning: explicit link request to 'endif' could not be resolved
sys/psa_crypto/doc.md:1007: warning: Unsupported xml/html tag <Some> found
As far as I can tell, the code is in valid code blocks and Doxygen 1.9.1 parses it correctly and so does the GitHub markdown preview.
Something seems to make Doxygen unhappy after around line 460-ish. But that's just one of more examples that have to be fixed before this PR will be freed by the static-test. OR I'd have to extend the exception list.
As far as I can tell, the code is in valid code blocks
It was not, see https://github.com/RIOT-OS/RIOT/pull/21380
Doxygen 1.9.1 parses it correctly and so does the GitHub markdown preview.
there is indeed a change in how doxygen parses it between 1.9.4 and 1.13.4, but the root cause is indeed the missing end of block.
With #21380, these are the warnings that remain:
cbuec@W11nMate:~/RIOTstuff/riot-psa-crypto-doc/RIOT$ ./dist/tools/doccheck/check.sh
ERROR: Doxygen generates the following warnings:
warning: Tag 'OUTPUT_TEXT_DIRECTION' at line 102 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'HTML_TIMESTAMP' at line 1366 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'FORMULA_TRANSPARENT' at line 1671 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'LATEX_SOURCE_CODE' at line 1984 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'LATEX_TIMESTAMP' at line 2000 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'RTF_SOURCE_CODE' at line 2074 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOCBOOK_PROGRAMLISTING' at line 2179 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'CLASS_DIAGRAMS' at line 2387 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOT_FONTNAME' at line 2429 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOT_FONTSIZE' at line 2436 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'DOT_TRANSPARENT' at line 2684 of file '-' has become obsolete.
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
sys/include/net/gnrc/netif/pktq.h:14: warning: Potential recursion while resolving \ref command!
cpu/rpx0xx/include/periph_cpu.h:234: warning: unable to resolve reference to 'gpio_t' for \ref command
sys/include/net/gnrc/netif/pktq/type.h:13: warning: Potential recursion while resolving \ref command!
boards/common/arduino-mkr/include/periph_conf_common.h:75: warning: Member CLOCK_DIV (macro definition) of file periph_conf_common.h is not documented.
boards/common/native/include/board.h:143: warning: Member SPIFFS_MTD_DEV (macro definition) of file board.h is not documented.
cpu/rpx0xx/include/periph_cpu.h:234: warning: unable to resolve reference to 'gpio_t' for \ref command
cpu/rpx0xx/include/periph_cpu.h:234: warning: unable to resolve reference to 'gpio_t' for \ref command
boards/arduino-nano-33-iot/include/periph_conf.h:69: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/common/arduino-zero/include/periph_conf.h:77: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/feather-m0/include/periph_conf.h:71: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/hamilton/include/periph_conf.h:84: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/samd10-xmini/include/periph_conf.h:88: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/samd20-xpro/include/periph_conf.h:90: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/samd21-xpro/include/periph_conf.h:90: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/samr21-xpro/include/periph_conf.h:90: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/seeeduino_xiao/include/periph_conf.h:73: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/sensebox_samd21/include/periph_conf.h:72: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
boards/serpente/include/periph_conf.h:71: warning: Member CLOCK_DIV (macro definition) of file periph_conf.h is not documented.
cpu/cortexm_common/include/cpu_conf_common.h:96: warning: unable to resolve reference to 'CPU_DEFAULT_IRQ_PRIO' for \ref command
boards/common/sodaq/include/cfg_clock_default.h:72: warning: Member CLOCK_DIV (macro definition) of file cfg_clock_default.h is not documented.
sys/include/ztimer/config.h:182: warning: unable to resolve reference to 'CONFIG_ZTIMER_USEC_ADJUST_SET' for \ref command
sys/include/net/gnrc/netif/pktq.h:14: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq.h:14: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq/type.h:13: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq/type.h:13: warning: Potential recursion while resolving \ref command!
sys/include/string_utils.h:37: warning: Member STRING_UTILS_H (macro definition) of group sys_string_utils is not documented.
doc/doxygen/src/porting-boards.md:53: warning: unable to resolve reference to 'XTIMER_BACKOFF' for \ref command
doc/doxygen/src/porting-boards.md:53: warning: unable to resolve reference to 'XTIMER_BACKOFF' for \ref command
sys/include/net/sock/dodtls.h:136: warning: unable to resolve reference to 'sock_dtls_create()' for \ref command
sys/include/net/sock/dtls.h:34: warning: unable to resolve reference to 'sock_dtls_create()' for \ref command
sys/include/net/sock/dtls.h:44: warning: unable to resolve reference to 'sock_dtls_create()' for \ref command
sys/include/net/sock/dtls.h:263: warning: unable to resolve reference to 'sock_dtls_create()' for \ref command
sys/include/net/sock/dtls.h:429: warning: unable to resolve reference to 'sock_dtls_close()' for \ref command
sys/include/net/sock/dtls.h:1108: warning: unable to resolve reference to 'sock_dtls_create()' for \ref command
sys/include/net/gnrc/pktbuf.h:60: warning: unable to resolve reference to 'CONFIG_GNRC_PKTBUF_SIZE' for \ref command
sys/include/net/gnrc.h:107: warning: unable to resolve reference to 'net_gnrc_netapi::GNRC_NETAPI_MSG_TYPE_RCV' for \ref command
boards/esp32-heltec-lora32-v2/doc.txt:81: warning: unable to resolve reference to 'PWM0_GPIOS' for \ref command
boards/msbiot/doc.txt:71: warning: unable to resolve reference to 'LED0_PIN' for \ref command
boards/msbiot/doc.txt:71: warning: unable to resolve reference to 'LED1_PIN' for \ref command
boards/msbiot/doc.txt:72: warning: unable to resolve reference to 'LED0_ON' for \ref command
boards/msbiot/doc.txt:72: warning: unable to resolve reference to 'LED1_ON' for \ref command
boards/msbiot/doc.txt:75: warning: unable to resolve reference to 'LED0_ON' for \ref command
boards/msbiot/doc.txt:75: warning: unable to resolve reference to 'LED0_OFF' for \ref command
boards/msbiot/doc.txt:75: warning: unable to resolve reference to 'LED0_TOGGLE' for \ref command
boards/msbiot/doc.txt:76: warning: unable to resolve reference to 'LED1_ON' for \ref command
boards/msbiot/doc.txt:76: warning: unable to resolve reference to 'LED1_OFF' for \ref command
boards/msbiot/doc.txt:76: warning: unable to resolve reference to 'LED1_TOGGLE' for \ref command
sys/posix/include/sys/socket.h:464: warning: The following parameter of socket(int domain, int type, int protocol) is not documented:
sys/include/net/gnrc/pktbuf.h:19: warning: unable to resolve reference to 'CONFIG_GNRC_PKTBUF_SIZE' for \ref command
drivers/include/rtt_rtc.h:49: warning: unable to resolve reference to 'RTT_FREQUENCY' for \ref command
drivers/include/rtt_rtc.h:36: warning: unable to resolve reference to 'RTT_FREQUENCY' for \ref command
sys/include/net/gnrc/netif/pktq/type.h:10: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq/type.h:10: warning: Potential recursion while resolving \ref command!
boards/esp32-ttgo-t-beam/doc.txt:82: warning: unable to resolve reference to 'PWM0_GPIOS' for \ref command
sys/include/string_utils.h:37: warning: Member STRING_UTILS_H (macro definition) of group sys_string_utils is not documented.
boards/esp32s2-wemos-mini/doc.txt:45: warning: unable to resolve reference to 'esp32s2-wemos-mini_toc' for \ref command
drivers/include/ph_oem.h:179: warning: unable to resolve reference to 'GPIO_IN_PU' for \ref command
drivers/include/ph_oem.h:180: warning: unable to resolve reference to 'GPIO_IN_PD' for \ref command
drivers/include/ph_oem.h:181: warning: unable to resolve reference to 'GPIO_IN_PD' for \ref command
sys/include/net/gnrc/netif.h:216: warning: Potential recursion while resolving \ref command!
sys/include/net/ieee802154_security.h:70: warning: unable to resolve reference to 'ieee802154_sec_context_t::ieee802154_sec_dev_t' for \ref command
sys/include/net/ieee802154_security.h:84: warning: unable to resolve reference to 'ieee802154_sec_context_t::ieee802154_sec_dev_t' for \ref command
sys/include/net/ieee802154_security.h:60: warning: unable to resolve reference to 'ieee802154_sec_context_t::ieee802154_sec_dev_t' for \ref command
<sys/include/net/gnrc/netif/>:1: warning: Potential recursion while resolving \ref command!
<sys/include/net/gnrc/netif/pktq/>:1: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq.h:14: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq/type.h:13: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq/type.h:13: warning: Potential recursion while resolving \ref command!
sys/include/net/gnrc/netif/pktq.h:14: warning: Potential recursion while resolving \ref command!
The first block of warnings would be solved with #21292, the others would require exceptions or have to be fixed...
Phew that was close. I didn't know that Automerge was still enabled from 7 months ago 🤣
Okay, I included the update to Doxygen 1.15.0 in this PR, because otherwise we have a hen-and-egg problem with the static test.
Every Doxygen update brings new warnings, so I had to add some to the exclude_simple file. They really should be fixed, but that is beyond the scope of this PR tbh.
We should probably make use of dlcache for the Doxygen archive, otherweise our CI will download Doxygen for eeeevery build, adding needless traffic and time.
Edit: Done.
Sorry for all the force-pushing, I fixed the tab-space-indentations in two files and disabled the Shellcheck warning for github_annotate.sh.
Though, why don't we simply make doc-ci the default for
make doc?
Maybe later, I didn't want to force everyone to download Doxygen if they already have it locally. Also there is a warning if the local Doxygen version is too old that points to doc-ci.