pytest-sphinx icon indicating copy to clipboard operation
pytest-sphinx copied to clipboard

`pytest-sphinx` not finding all tests that `sphinx-build -b doctest` does

Open cr1901 opened this issue 10 months ago • 0 comments

First of all, thank you for writing pytest-sphinx. I find it much nicer to use than sphinx-build.

I use pytest-sphinx in a project to good effect. However, past-me wrote a comment that current-me can't quite figure out:

# FIXME: pytest-sphinx doesn't catch all of these. Remove when it does.
doc-test = "sphinx-build -b doctest doc/ doc/_build"

Indeed, when I run pytest, it's missing detecting code in my usage.md that sphinx-build detects (same result without any -k filters):

sphinx-build

William@DESKTOP-3H1DSBV MINGW64 ~/Projects/FPGA/amaranth/smolarith
$ pdm run sphinx-build -b doctest doc/ doc/_build
Running Sphinx v7.1.2
loading pickled environment... done
myst v2.0.0: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=set(), disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, fence_as_directive=set(), number_code_blocks=[], title_to_header=False, heading_anchors=3, heading_slug_func=None, html_meta={}, footnote_transition=False, words_per_minute=200, substitutions={}, linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', enable_checkboxes=False, suppress_warnings=[], highlight_code_blocks=True)
building [mo]: targets for 0 po files that are out of date
writing output...
building [doctest]: targets for 6 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources...
looking for now-outdated files... none found
running tests...

Document: usage
---------------
1 items passed all tests:
   1 tests in default
1 tests in 1 items.
1 passed and 0 failed.
Test passed.

Document: impl
--------------
1 items passed all tests:
   4 tests in default
4 tests in 1 items.
4 passed and 0 failed.
Test passed.

Document: api
-------------
1 items passed all tests:
   7 tests in default
7 tests in 1 items.
7 passed and 0 failed.
Test passed.

Doctest summary
===============
   12 tests
    0 failures in tests
    0 failures in setup code
    0 failures in cleanup code
build succeeded.

Testing of doctests in the sources finished, look at the results in doc/_build/output.txt.

pytest-sphinx

William@DESKTOP-3H1DSBV MINGW64 ~/Projects/FPGA/amaranth/smolarith
$ pdm run pytest --doctest-modules --ignore=examples -k "doc or output_signature" --co
======================================================== test session starts ========================================================= platform win32 -- Python 3.11.9, pytest-8.1.1, pluggy-1.4.0
rootdir: C:/msys64/home/William/Projects/FPGA/amaranth/smolarith
configfile: pyproject.toml
plugins: sphinx-0.6.3, xdist-3.5.0
collected 69 items / 65 deselected / 4 selected

<Dir smolarith>
  <Dir doc>
    <DoctestTextfile api.md>
      <DoctestItem api.md>
    <DoctestTextfile impl.md>
      <DoctestItem impl.md>
  <Dir src>
    <Package smolarith>
      <DoctestModule div.py>
        <DoctestItem smolarith.div.divider_output_signature>
      <DoctestModule mul.py>
        <DoctestItem smolarith.mul.multiplier_output_signature>

Is it possible to ensure that usage.md doctests are detected, and is there a way to trace pytest-sphinx and determine whether it's missing more tests, such as "some but not all doctests detected within a file"?

cr1901 avatar Apr 16 '24 18:04 cr1901