Doctests as testitem?

[gdalle]

I'm not sure how to run doctests as a testitem:

• the output of doctest(MyPackage) is not a bool, so I can't do @test doctest(MyPackage) inside the @testitem
• when I do doctest(MyPackage) directly inside the @testitem, it fails with an uninformative error message:

Test Failed at ~.julia/packages/Documenter/H5y27/src/Documenter.jl:963
  Expression: all_doctests()

[gdalle]

By digging around in the output I found some more information. It probably has to do with the execution of the testitems in a new module:

[ Info: Doctest: running doctests.
┌ Warning: Failed to evaluate `CurrentModule = MyPackage` in `@meta` block.
│   exception = UndefVarError: `MyPackage` not defined
└ @ Documenter.DocTests ~/.julia/packages/Documenter/H5y27/src/Utilities/Utilities.jl:34
┌ Error: Doctesting failed
│   exception =
│    `makedocs` encountered a doctest error. Terminating build

[davidanthoff]

We should probably just try to detect doc tests statically and then treat them as a special kind of test item, or something like that...

[ORBAT]

This is the workaround I've been using to run doctests in a @testitem. It temporarily adds the standard environment "@v#.#" to the load path so that my package doesn't have to have Documenter in its dependencies (ie. it assumes that Documenter can be found in that environment), loads Documenter and then runs the actual function that calls doctest. Fooling with the load path while "live" is probably not a great idea, but I haven't personally run into any surprises yet.

I feel like there's got to be better ways to do this, but I'm a Julia beginner so this is the best I could come up with 😅

using TestItems

@testitem "doctests" begin
  function with_documenter(fn)
    env = "@v#.#"

    if !(env in Base.LOAD_PATH)
      insert!(Base.LOAD_PATH, 2, env)
      @info "changed LOAD_PATH" Base.LOAD_PATH
    end

    __mod = @__MODULE__()
    @eval __mod using Documenter

    try
      @info "running test fn"
      @eval __mod $fn()
      @info "test fn done"
    catch e
      rethrow(e)
    finally
      let idx = findfirst(isequal(env), Base.LOAD_PATH)
        if idx == nothing
          return
        end
        deleteat!(Base.LOAD_PATH, idx)
        @info "reset LOAD_PATH" Base.LOAD_PATH
      end
    end
  end

  with_documenter() do
    DocMeta.setdocmeta!(Musica, :DocTestSetup, :(using Musica); recursive=true)
    doctest(Musica; manual=false)
  end
end