Use pytest assertion rewriting

[gartentrio]

Is there a way to make use of pytest's pretty assertion rewriting with nbval?

[vidartf]

Output comparison should use pytest's pretty assert repr since 0.8. See this code. If you're on a version after that, and are not seeing this, would you mind posting the (minimal) steps to reproduce that behavior?

[gartentrio]

Test code

We are using nbval 0.9.0.

a = 'dog'
assert a == 'cat'

pytest with pretty assertions

pytest with nbval without pretty assertions

[vidartf]

So this would be using pytest for assert statements in the notebook. I'm not sure if think this is a good idea or not. It would require to special case Python kernels, and to run extra code on the kernel, meaning it should at the very least be behind a flag.

For now I would recommend you add a conf.py file in the folder of your notebooks, and use one of the collection hooks to run the needed code on the kernel. I don't know how pytest sets up its smarter assertion code for python, so I can't help you there. If you get it working, I'd be happy to help review a PR!

[chmp]

You can use ipytest to setup assertion rewriting via AST transforms. This way the left and right hand side of the equal sign will be included in the output. However it will still not include the diffing output, you would get when running in pytest proper. (ipytest will also include this output, but currently swallows any errors so nbval would not show an error). For example:

# cell 0
import ipytest
ipytest.config.rewrite_asserts = True

# cell 1
a = 'dog'
assert a == 'cat'

# output using nbval:
# AssertionError                            Traceback (most recent call last)
# <ipython-input-2-2e3c346211ed> in <module>
#       1 a = 'dog'
# ----> 2 assert a == 'cat'
# 
# AssertionError: assert 'dog' == 'cat'

Not sure, whether you would like to include such functionality in nbval. If so, I am happy to extract the relevant code from ipytest and create a PR. However, I think there is a value in putting the setup of the ast rewriting into the notebook itself and not to include it into nbval to ensure you always run the same code within the notebook frontend and nbval.

[vidartf]

@chmp Thanks! It makes sense to me to have a different package for things that you use within a python notebook, compared to nbval which is used agnostically on a notebook.

[...] to ensure you always run the same code within the notebook frontend and nbval.

👍