Feature request: Check if objects documented as expected

[RaphaelS1]

Hey I'm relatively new to Julia and Documenter.jl so apologies if this has already been discussed or is in fact available (though couldn't see when scrolling through issues). I've come from R where roxygen2+pkgdown is very strict in checking every exported function is documented - this would be a great feature in Documenter.jl as it ensures everything is properly documented. I get the challenge here is that it's not easy to pick-out re-exports and aliases so the problem is non-trivial.

Anyway here is an example of how I mocked it up in my own package (apologies for sub-optimal Julia code):

1. I've added tags in the main project file to show which exported objects I expect to document
2. I've added a script that reads my compiled API using Gumbo+Cascadia to search through .docstring-binding code tags and then matches these against those objects in the 'documented' section from (1). If any objects are missed then there's a fatal error or if too many are documented there's a warning.
3. The script from (2) is called from a GitHub actions workflow

I appreciate this will be low on your plans but a more streamlined version that could work for any package would be amazing.

[RaphaelS1]

I closed then re-opened it due to discussion in this thread: https://twitter.com/RaphaelS101/status/1565612443589566464.

Basically it seemed as if checkdocs=:all does this but in my experiments it does not

[mortenpi]

Documenter checks if a given docstring is included in the generated manual, but it does not check whether every function/type/module has a docstring.

If I understand correctly, you're after the latter? Such a functionality doesn't really require Documenter. Also, I am not sure how useful it would be in general -- idiomatic Julia code often contains lots of small, internal functions and variables that are probably not worth documenting (exports are a different story though).

[RaphaelS1]

If I understand correctly, you're after the latter?

Yeah, or not really 'every' object but just ones we want to be documented (so if you look at point 1 in my original post you'll see an example)

Also, I am not sure how useful it would be in general -- idiomatic Julia code often contains lots of small, internal functions and variables that are probably not worth documenting (exports are a different story though).

I'm coming from R where if you use an @export tag then roxygen2 ensures you've also written a full docstring (which is automatically added to API). So that's kind of the analogue I'm thinking of

[mortenpi]

For checking exports, it should be relatively straightforward to write a small function that cross-checks what name(MyPackage) gives vs what is in Docs.meta(MyPackage).

And even though it doesn't really have much to do with Documenter per se, it wouldn't necessarily be out of place either here.

[RaphaelS1]

Docs.meta(MyPackage)

Thanks for this tip, much neater than in my code above.

name(MyPackage)

Assuming you meant names, if so this unfortunately doesn't quite work as it includes exported objects that shouldn't be documented or don't need to be, for example exported aliases or exports that trivially extend functions from other packages.

[mortenpi]

Assuming you meant names, if so this unfortunately doesn't quite work as it includes exported objects that shouldn't be documented or don't need to be, for example exported aliases or exports that trivially extend functions from other packages.

names, yes. But you'd indeed need to do some more introspection, to see where the binding is defined etc.

[RaphaelS1]

Deleted previous comment - pretty messy! Found a solution that doesn't require any additional dependencies, macros, or extra functions. Let me know your thoughts!

using MyPackage

documented = map(x -> replace(string(x), "MyPackage." => ""),
    collect(keys(Docs.meta(MyPackage))));

expected = [];
map(x ->
    x != :MyPackage && ## remove package name
    Symbol(eval(x)) == x && # remove aliases
    parentmodule(eval(x)) == MyPackage && ## check if this package is parent module
    push!(expected, strip(String(x))), names(MyPackage)); ## push if all true

missings = setdiff(expected, documented); ## under-documented, critical error
extras = setdiff(documented, expected); ## over-documented, warning

if length(missings) > 0
    error("Forgot to document: ", missings)
elseif length(extras) > 0
    @warn "Extra objects documented:" extras
else
    @info "All objects documented as expected:" expected
end

[mortenpi]

For the time being, I would suggest perhaps turning this into a separate tiny package, since it doesn't require anything from Documenter to work. It could then run either in the test/runtests.jl or in docs/make.jl, depending on how the user wants to use it.

It might be worth having it in Documenter in the future, but I don't have a clear vision of what scope we want for the documentation checks here right now, and so I am a little hesitant to increase the maintenance burden here.