MyST-Parser
MyST-Parser copied to clipboard
executablebooks
tests/test_sphinx/test_sphinx_builds.py::test_basic fails if pytest_param_files is not in the environment
Describe the bug
context
When I build MyST-Parser 0.18.0 for Fedora 37 with Python3.11~b3, Sphinx 5.0.2, docutils 0.18.1 I run a test suite with available components.
I disable tests requiring linkify and pytest_param_files, as we don't have those packages available.
That's pretty much the whole incantation:
pytest -k 'not test_extended_syntaxes' --ignore 'tests/test_renderers/test_fixtures_sphinx.py' --ignore 'tests/test_renderers/test_fixtures_docutils.py' --deselect 'tests/test_html/test_html_to_nodes.py::test_html_to_nodes' --deselect 'tests/test_html/test_parse_html.py::test_html_ast' --deselect 'tests/test_html/test_parse_html.py::test_html_round_trip' --deselect 'tests/test_renderers/test_error_reporting.py::test_basic' --deselect 'tests/test_renderers/test_include_directive.py::test_render' --deselect 'tests/test_renderers/test_include_directive.py::test_errors' --deselect 'tests/test_renderers/test_myst_config.py::test_cmdline' --deselect 'tests/test_renderers/test_parse_directives.py::test_parsing'
expectation I expected tests to pass.
bug
But instead one test fails. test_basic_sphinx which doesn't require linkify, nor pytest_param_files fixture.
It fails repeatedly with the same problem.
app = <SphinxTestApp buildername='html'>
status = <_io.StringIO object at 0x7fd9cfe05870>
warning = <_io.StringIO object at 0x7fd9cfe05a20>
get_sphinx_app_doctree = <function get_sphinx_app_doctree.<locals>.read at 0x7fd9cfcf94e0>
get_sphinx_app_output = <function get_sphinx_app_output.<locals>.read at 0x7fd9cfcf9620>
@pytest.mark.sphinx(
buildername="html",
srcdir=os.path.join(SOURCE_DIR, "basic"),
freshenv=True,
confoverrides={"myst_enable_extensions": ["dollarmath"]},
)
def test_basic(
app,
status,
warning,
get_sphinx_app_doctree,
get_sphinx_app_output,
):
"""basic test."""
app.build()
assert "build succeeded" in status.getvalue() # Build succeeded
warnings = warning.getvalue().strip()
assert warnings == ""
try:
> get_sphinx_app_doctree(
app,
docname="content",
regress=True,
)
tests/test_sphinx/test_sphinx_builds.py:41:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
app = <SphinxTestApp buildername='html'>, docname = 'content', resolve = False
regress = True, replace = None, regress_ext = '.xml'
def read(
app,
docname="index",
resolve=False,
regress=False,
replace=None,
regress_ext=".xml",
):
if resolve:
doctree = app.env.get_and_resolve_doctree(docname, app.builder)
extension = f".resolved{regress_ext}"
else:
doctree = app.env.get_doctree(docname)
extension = regress_ext
# convert absolute filenames
for node in findall(doctree)(
lambda n: "source" in n and not isinstance(n, str)
):
node["source"] = pathlib.Path(node["source"]).name
if regress:
text = doctree.pformat() # type: str
for find, rep in (replace or {}).items():
text = text.replace(find, rep)
> file_regression.check(text, extension=extension)
tests/test_sphinx/conftest.py:123:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pytest_regressions.file_regression.FileRegressionFixture object at 0x7fd9cfcaa3d0>
contents = '<document source="content.md">\n <topic classes="dedication">\n <title>\n Dedication\n <p...tution references:\n <paragraph>\n 57\n words | \n 0\n min read\n'
encoding = None, extension = '.xml', newline = None, basename = None
fullpath = None, binary = False, obtained_filename = None
check_fn = functools.partial(<function check_text_files at 0x7fd9d03aa660>, encoding=None)
def check(
self,
contents,
encoding=None,
extension=".txt",
newline=None,
basename=None,
fullpath=None,
binary=False,
obtained_filename=None,
check_fn=None,
):
"""
Checks the contents against a previously recorded version, or generate a new file.
:param str contents: content to be verified.
:param str|None encoding: Encoding used to write file, if any.
:param str extension: Extension of file.
:param str|None newline: See `io.open` docs.
:param bool binary: If the file is binary or text.
:param obtained_filename: ..see:: FileRegressionCheck
:param check_fn: a function with signature ``(obtained_filename, expected_filename)`` that should raise
AssertionError if both files differ.
If not given, use internal function which compares text using :py:mod:`difflib`.
"""
__tracebackhide__ = True
if binary and encoding:
raise ValueError(
"Only binary ({!r}) or encoding ({!r}) parameters must be passed at the same time.".format(
binary, encoding
)
)
if binary:
assert isinstance(
contents, bytes
), "Expected bytes contents but received type {}".format(
type(contents).__name__
)
else:
assert isinstance(
contents, str
), "Expected text/unicode contents but received type {}".format(
type(contents).__name__
)
import io
if check_fn is None:
if binary:
def check_fn(obtained_filename, expected_filename):
if obtained_filename.read_bytes() != expected_filename.read_bytes():
raise AssertionError(
"Binary files {} and {} differ.".format(
obtained_filename, expected_filename
)
)
else:
check_fn = partial(check_text_files, encoding=encoding)
def dump_fn(filename):
mode = "wb" if binary else "w"
with open(str(filename), mode, encoding=encoding, newline=newline) as f:
f.write(contents)
> perform_regression_check(
datadir=self.datadir,
original_datadir=self.original_datadir,
request=self.request,
check_fn=check_fn,
dump_fn=dump_fn,
extension=extension,
basename=basename,
fullpath=fullpath,
force_regen=self.force_regen,
with_test_class_names=self.with_test_class_names,
obtained_filename=obtained_filename,
)
/usr/lib/python3.11/site-packages/pytest_regressions/file_regression.py:93:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
datadir = PosixPath('/tmp/pytest-of-mockbuild/pytest-0/test_basic0/test_sphinx_builds')
original_datadir = PosixPath('/builddir/build/BUILD/MyST-Parser-0.18.0/tests/test_sphinx/test_sphinx_builds')
request = <SubRequest 'file_regression' for <Function test_basic>>
check_fn = functools.partial(<function check_text_files at 0x7fd9d03aa660>, encoding=None)
dump_fn = <function FileRegressionFixture.check.<locals>.dump_fn at 0x7fd9cfbb2f20>
extension = '.xml', basename = 'test_basic', fullpath = None
force_regen = False, with_test_class_names = False
obtained_filename = PosixPath('/tmp/pytest-of-mockbuild/pytest-0/test_basic0/test_sphinx_builds/test_basic.obtained.xml')
dump_aux_fn = <function <lambda> at 0x7fd9d03aa7a0>
def perform_regression_check(
datadir,
original_datadir,
request,
check_fn,
dump_fn,
extension,
basename=None,
fullpath=None,
force_regen=False,
with_test_class_names=False,
obtained_filename=None,
dump_aux_fn=lambda filename: [],
):
"""
First run of this check will generate a expected file. Following attempts will always try to
match obtained files with that expected file.
If expected file needs to be updated, just enable `force_regen` argument.
:param Path datadir: Fixture embed_data.
:param Path original_datadir: Fixture embed_data.
:param SubRequest request: Pytest request object.
:param callable check_fn: A function that receives as arguments, respectively, absolute path to
obtained file and absolute path to expected file. It must assert if contents of file match.
Function can safely assume that obtained file is already dumped and only care about
comparison.
:param callable dump_fn: A function that receive an absolute file path as argument. Implementor
must dump file in this path.
:param callable dump_aux_fn: A function that receives the same file path as ``dump_fn``, but may
dump additional files to help diagnose this regression later (for example dumping image of
3d views and plots to compare later). Must return the list of file names written (used to display).
:param str extension: Extension of files compared by this check.
:param bool force_regen: if true it will regenerate expected file.
:param bool with_test_class_names: if true it will use the test class name (if any) to compose
the basename.
:param str obtained_filename: complete path to use to write the obtained file. By
default will prepend `.obtained` before the file extension.
..see: `data_regression.Check` for `basename` and `fullpath` arguments.
"""
import re
assert not (basename and fullpath), "pass either basename or fullpath, but not both"
__tracebackhide__ = True
with_test_class_names = with_test_class_names or request.config.getoption(
"with_test_class_names"
)
if basename is None:
if (request.node.cls is not None) and (with_test_class_names):
basename = re.sub(r"[\W]", "_", request.node.cls.__name__) + "_"
else:
basename = ""
basename += re.sub(r"[\W]", "_", request.node.name)
if fullpath:
filename = source_filename = Path(fullpath)
else:
filename = datadir / (basename + extension)
source_filename = original_datadir / (basename + extension)
def make_location_message(banner, filename, aux_files):
msg = [banner, f"- {filename}"]
if aux_files:
msg.append("Auxiliary:")
msg += [f"- {x}" for x in aux_files]
return "\n".join(msg)
force_regen = force_regen or request.config.getoption("force_regen")
if not filename.is_file():
source_filename.parent.mkdir(parents=True, exist_ok=True)
dump_fn(source_filename)
aux_created = dump_aux_fn(source_filename)
msg = make_location_message(
"File not found in data directory, created:", source_filename, aux_created
)
pytest.fail(msg)
else:
if obtained_filename is None:
if fullpath:
obtained_filename = (datadir / basename).with_suffix(
".obtained" + extension
)
else:
obtained_filename = filename.with_suffix(".obtained" + extension)
dump_fn(obtained_filename)
try:
> check_fn(obtained_filename, filename)
/usr/lib/python3.11/site-packages/pytest_regressions/common.py:162:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
obtained_fn = PosixPath('/tmp/pytest-of-mockbuild/pytest-0/test_basic0/test_sphinx_builds/test_basic.obtained.xml')
expected_fn = PosixPath('/tmp/pytest-of-mockbuild/pytest-0/test_basic0/test_sphinx_builds/test_basic.xml')
fix_callback = <function <lambda> at 0x7fd9d03aa520>, encoding = None
def check_text_files(obtained_fn, expected_fn, fix_callback=lambda x: x, encoding=None):
"""
Compare two files contents. If the files differ, show the diff and write a nice HTML
diff file into the data directory.
:param Path obtained_fn: path to obtained file during current testing.
:param Path expected_fn: path to the expected file, obtained from previous testing.
:param str encoding: encoding used to open the files.
:param callable fix_callback:
A callback to "fix" the contents of the obtained (first) file.
This callback receives a list of strings (lines) and must also return a list of lines,
changed as needed.
The resulting lines will be used to compare with the contents of expected_fn.
"""
__tracebackhide__ = True
obtained_fn = Path(obtained_fn)
expected_fn = Path(expected_fn)
obtained_lines = fix_callback(obtained_fn.read_text(encoding=encoding).splitlines())
expected_lines = expected_fn.read_text(encoding=encoding).splitlines()
if obtained_lines != expected_lines:
diff_lines = list(
difflib.unified_diff(expected_lines, obtained_lines, lineterm="")
)
if len(diff_lines) <= 500:
html_fn = obtained_fn.with_suffix(".diff.html")
try:
differ = difflib.HtmlDiff()
html_diff = differ.make_file(
fromlines=expected_lines,
fromdesc=expected_fn,
tolines=obtained_lines,
todesc=obtained_fn,
)
except Exception as e:
html_fn = "(failed to generate html diff: %s)" % e
else:
html_fn.write_text(html_diff, encoding="UTF-8")
diff = ["FILES DIFFER:", str(expected_fn), str(obtained_fn)]
diff += ["HTML DIFF: %s" % html_fn]
diff += diff_lines
> raise AssertionError("\n".join(diff))
E AssertionError: FILES DIFFER:
E /tmp/pytest-of-mockbuild/pytest-0/test_basic0/test_sphinx_builds/test_basic.xml
E /tmp/pytest-of-mockbuild/pytest-0/test_basic0/test_sphinx_builds/test_basic.obtained.xml
E HTML DIFF: /tmp/pytest-of-mockbuild/pytest-0/test_basic0/test_sphinx_builds/test_basic.obtained.diff.html
E ---
E +++
E @@ -53,7 +53,7 @@
E https://www.google.com
E <paragraph>
E <strong>
E - <literal classes="code">
E + <literal classes="code" language="">
E a=1{`}
E <paragraph>
E <math>
/usr/lib/python3.11/site-packages/pytest_regressions/common.py:57: AssertionError
During handling of the above exception, another exception occurred:
<snip - this is the same output for the `finally` block there>
tests/test_sphinx/conftest.py:123: AssertionError
--------------------------- Captured stdout teardown ---------------------------
# testroot: root
# builder: html
# srcdir: /builddir/build/BUILD/MyST-Parser-0.18.0/tests/test_sphinx/sourcedirs/basic
# outdir: /builddir/build/BUILD/MyST-Parser-0.18.0/tests/test_sphinx/sourcedirs/basic/_build/html
# status:
Running Sphinx v5.0.2
myst v0.18.0: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=['dollarmath'], disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, footnote_transition=True, words_per_minute=200, sub_delimiters=('{', '}'), linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area')
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 3 source files that are out of date
updating environment: [new config] 3 added, 0 changed, 0 removed
reading sources... [100%] orphan
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] orphan
generating indices... genindex done
writing additional pages... search done
copying images... [100%] example.jpg
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
The HTML pages are in tests/test_sphinx/sourcedirs/basic/_build/html.
problem
I packaged pytest_param_files as RPM and ran all the tests (except those which rely on linkify). This time the test suite passed.
But here's the thing, AFAIK it shouldn't be needed for this particular test to pass. It's never used in test_sphinx_builds.py and conftest.py. The problem in the pasted output doesn't seem connected to that small fixture at all. Yet when it is present and I let all the test run, something in the environment changes so that the output doesn't produce the additional language="" string.
At this point of investigation I don't know what I ran into - whether my original setup was botched and failure could be expected or there's a bug in test setup that makes the environment fragile, or maybe even a problem in production code that's kind of shadowed by the tests. I don't really know how to proceed from here, except I'll add pytest_param_files to Fedora and hope for the best.
Reproduce the bug
In a Python3.11~b3 venv (works in Python 3.10 too), set up the test environment for MyST-Parser
$ python -m pip install --upgrade pip
# The problem occurs with the current master version too
$ pip install -e .[testing]
# Just make sure the correct versions are installed & not installed
$ pip install sphinx==5.0.2 docutils==0.18.1 pytest==7.1.2
$ pip uninstall pytest-param-files linkify-it-py
$ python -m pytest -k 'not test_extended_syntaxes' --ignore 'tests/test_renderers/test_fixtures_sphinx.py' --ignore 'tests/test_renderers/test_fixtures_docutils.py' --deselect 'tests/test_html/test_html_to_nodes.py::test_html_to_nodes' --deselect 'tests/test_html/test_parse_html.py::test_html_ast' --deselect 'tests/test_html/test_parse_html.py::test_html_round_trip' --deselect 'tests/test_renderers/test_error_reporting.py::test_basic' --deselect 'tests/test_renderers/test_include_directive.py::test_render' --deselect 'tests/test_renderers/test_include_directive.py::test_errors' --deselect 'tests/test_renderers/test_myst_config.py::test_cmdline' --deselect 'tests/test_renderers/test_parse_directives.py::test_parsing'
### List your environment
Fedora 35-37, MyST-Parser 0.18.0 (and current master), Python 3.10.5 & 3.11~beta3
Python 3.10.5 venv contents
$ pip freeze alabaster==0.7.12 attrs==21.4.0 Babel==2.10.3 beautifulsoup4==4.11.1 certifi==2022.6.15 charset-normalizer==2.0.12 coverage==6.4.1 docutils==0.18.1 idna==3.3 imagesize==1.3.0 iniconfig==1.1.1 Jinja2==3.1.2 markdown-it-py==2.1.0 MarkupSafe==2.1.1 mdit-py-plugins==0.3.0 mdurl==0.1.1 -e git+ssh://[email protected]/executablebooks/MyST-Parser.git@75ef9cb7b65c98d969724fc6c096e8d6209c5ea0#egg=myst_parser packaging==21.3 pluggy==1.0.0 py==1.11.0 Pygments==2.12.0 pyparsing==3.0.9 pytest==7.1.2 pytest-cov==3.0.0 pytest-datadir==1.3.1 pytest-regressions==2.3.1 pytz==2022.1 PyYAML==6.0 requests==2.28.0 snowballstemmer==2.2.0 soupsieve==2.3.2.post1 Sphinx==5.0.2 sphinx_pytest==0.0.4 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 toml==0.10.2 tomli==2.0.1 typing_extensions==4.2.0 urllib3==1.26.9
Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:
![welcome[bot] avatar](https://avatars.githubusercontent.com/in/4141?v=4 "Published by a pub.dev verified publisher"){kind=small}
Heya, FYI this appears to be nothing to do with pytest_param_files per se.
I can replicate the same issue, by simply running: tox tests/test_sphinx/test_sphinx_builds.py::test_basic
So it appears that it is the order of testing that does it, i.e. something in a previous test is changing a global variable or something
So it appears to be something to do with the docutils code role, if you run either of these tests, then things change:
tests/test_renderers/fixtures/docutil_roles.md:
111 --------------------------------
112 [code] (`docutils.parsers.rst.roles.code_role`):
113 .
114: {code}`a`
115 .
116 <document source="notset">
117 <paragraph>
tests/test_renderers/fixtures/docutil_syntax_elements.md:
287
288 Sphinx Role containing backtick:
289 .
290: {code}``a=1{`}``
291 .
292 <document source="notset">
293 <paragraph>
I'm experiencing a similar issue on Windows 10:
OS: Windows 10, 21H2
Python: 3.8.10, x64-bit
MyST-Parser: 0.18.1
I followed the Contributing steps and installed linkify-py-it, but am getting 2 errors when I run pytest
======================================== short test summary info ========================================
FAILED tests/test_sphinx/test_sphinx_builds.py::test_gettext_html - AssertionError: FILES DIFFER:
FAILED tests/test_sphinx/test_sphinx_builds.py::test_fieldlist_extension - AssertionError: FILES DIFFER:
=============================== 2 failed, 999 passed, 14 skipped in 17.25s ==============================
Attached is an HTML report of the test run using pytest-html:
@chrisjsewell These fail consistently for python 3.7-3.10 and sphinx4 and 5, tested using tox. It's always:
test_fieldlist_extension.obtained.diff.htmltest_gettext_html.obtained.diff.html
@chrisjsewell These are also failing in your most recent CI, so there is definitely something awry here...
