WIP: Docutils parser

[tony]

Implements #37

What's incomplete

• Tests
• Potentially:
  • pytest reporting / line numbers
  • pytest doctest may be causing duplicate test items

What's supported

• reStructuredText: Pure docutils parsing

• Markdown: via myst-parser (github) + docutils

• docutils' style doctest blocks

  >>> print "This is a doctest block."
  This is a doctest block.
  

  docutils.nodes.doctest_block: docutils documentation, on pydoc.dev

• .. doctest [group] blocks with no options

  reStructuredText:

  .. doctest::
  
      >>> 2 + 2
      4
  
  .. doctest:: Our example
  
      >>> round(
      ...     3.9
      ... )
      4
  
  .. doctest:: Here's a function
  
      >>> def shout(message: str) -> str:
      ...     print(f'{message.upper()}')
  
      >>> shout('hello')
      HELLO
  

  markdown:

  Another:
  
  ```{doctest} My example
  >>> 2 + 3
  5
  ```
  
  ```{doctest} A second
  >>> 1 + 8
  9
  >>> 2 - 8
  -6
  ```
  

What's removed

• Removed sphinx.ext.doctest's support for:

  • .. testsetup:: [group]

  • .. testcleanup:: [group]

  • .. testcode:: [group]

  • .. testoutput:: [group]

  • Options for .. doctest:: [group]: These could potentially be added back

Requirements

$ pip install docutils myst-parser

Test rerunning

Doctest: install entr(1)

Pytest: install pytest-watcher

$ pip install pytest-watcher

doctest module

Normal runs

Per doctest, these are silent unless an error is detected:

There’s no output! That’s normal, and it means all the examples worked. Pass -v to the script, and doctest prints a detailed log of what it’s trying, and prints a summary at the end

reStructuredText:

python -m doctest_docutils examples/test.rst

Markdown:

python -m doctest_docutils examples/test.md

Multiple files:

python -m doctest_docutils examples/test.rst examples/test.md

-v: Show results

These will output even if there's no error

python -m doctest_docutils examples/test.rst examples/test.md -v

Verbose logging

python -m doctest_docutils examples/test.rst -v --log-level DEBUG

Retry on file save

via entr(1), requires separate installation:

ls *.py examples/*.* tests/*.py | entr -s 'python -m doctest_docutils examples/test.rst examples/test.md -v'

pytest

$ pytest examples/

Output:

==== test session starts =====
platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: ~/python/pytest-sphinx
plugins: sphinx-0.4.0
collected 9 items

examples/test.md ...                                       [ 33%]
examples/test.rst ....                                     [ 77%]
examples/test.txt ..                                       [100%]

===== 9 passed in 0.10s ======

Verbose

$ pytest examples/test.rst -v

Output:

==== test session starts =====
platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: ~/python/pytest-sphinx
plugins: sphinx-0.4.0
collected 5 items

examples/test.rst::Here's a function PASSED    [ 20%]
examples/test.rst::Our example PASSED          [ 40%]
examples/test.rst::test.rst[0] PASSED          [ 60%]
examples/test.rst::test.rst[1] PASSED          [ 80%]
examples/test.rst::test.rst PASSED             [100%]

===== 5 passed in 0.03s ======

[tony]

@thisch If you check out the PR, do pip install -e '.[lint,myst-parser]'

Do the example commands above work?

e.g. pytest examples/, python -m doctest_docutils examples/test.rst -v

You can also try putting errors into examples/ and running them.

[twmr]

Hi! Thx a lot for putting so much work into this!

Let me first try to test your included examples/ without pytest-sphinx. I'd like to understand what the problems with the doctest pytest plugin are. So maybe it makes sense to upstream your code into pytest.

edit: So here are my first results (Forgive me if you have already mentioned the differences somewhere):

• pytest examples/ by default doesn't search for .rst and .md unless the --doctest-glob option is specified on the command line.
• If there are multiple doctest blocks inside e.g. a rst file, pytest only creates a single pytest.Item for that, i.e., if you run pytest on a single rst file you only see that one "test" was found, even though all doctests in the rst are executed.
• The doctests in the markdown file are not correctly detected by the pytest-doctest plugin.
• The group-names are not shown in the verbose output of pytest, because the doctest pytest plugin doesn't parse them.

@tony which library supports parsing the .. doctest:: directive? Is this supported by the docutils? If yes, where can I find it in the docs?

[tony]

@thisch in re: your tests here

Can you do git pull --rebase --autostash, wipe your virtualenv and reinstall dependencies? I did a lot of changes and want to make sure you caught them.

are not correctly detected by the pytest-doctest plugin.

In regards to this QA list when you are testing, can you show me when it errors / how it errors / etc? For me py.test examples/ just works and collects everywhere.

[twmr]

@tony Sry that my comment wasn't clear enough. What I did in my tests was just to test your examples (rst and markdown files in examples/) in a virtualenv, where neither your changes are installed nor pytest-sphinx is installed.

The reason why I did that is that I wanted to see what is already supported by pytest and what isn't.

[tony]

@tony which library supports parsing the .. doctest:: directive? Is this supported by the docutils? If yes, where can I find it in the docs?

The setup() in doctest_docutils adds it via directives.register_directive().

The directive is a compromise for not fully supported sphinx.ext.doctest, but it's nice because it helps support some sphinx users (as long as its not using options)

For context, docutils itself, only the raw node is, e.g.

>>> 4 + 4

[twmr]

The setup() in doctest_docutils adds it via directives.register_directive().

Okay, so this was added because both sphinx as well as pytest-sphinx support this directive already, but it is not an integral part of docutils.
I was asking this question because I'm drafting a reply to your github question in https://github.com/pytest-dev/pytest/discussions/10155

[tony]

@tony Sry that my comment wasn't clear enough. What I did in my tests was just to test your examples (rst and markdown files in examples/) in a virtualenv, where neither your changes are installed nor pytest-sphinx is installed.

Got it

P.S. Another rebase one the way (post-merge of #43 #45)

[tony]

@thisch Rebased at e22846f