nb_diff_ignore settings in pytest.ini are being ignored

[jgostick]

I am very excited about using this plugin. It does exactly what we need on our repo. We have a folder of jupyter notebook examples which we want to test to make sure they are up to data with changes in the main package. We point users to these examples so want to make sure they all work.

Anyway, I've got your plugin working fine with pytest, but only if I put several lines in the meta data of each notebook (eg. "nbreg": { "diff_ignore": [ ] } ). I would really rather not put data explicitly into each file, but rather put some generic statements in the pytest.ini file which apply to all notebooks. Specifically, ignoring metadata and not comparing figures, plus a few other things.

I think I followed the documentation, but nothing I put in the pytest.ini file seems to make any difference.

Here is the pytest-notebook section of my file:

nb_diff_ignore =
    /metadata
    /cells/*/execution_count
    /cells/*/outputs/*/data
    /cells/*/outputs/*/text
    /cells/*/outputs/*/data/image/png

Any suggestions?

[chrisjsewell]

Good to hear thanks 😀 Hmm, I can't see in the code where it would be specifically required that there should be metadata. Perhaps if you can provide me a minimal working example notebook, and drop it here as something like test.ipynb.txt, and also maybe paste here the output exception that you are getting, an/or what you were expecting to get.

[jgostick]

minimal_notebook_testing_example.zip

Above is a zip file containing the following:

-notebooks/minimal_notebook_1.ipnyb, minimal_notebook_2.ipnyb -tests/NotebookTest.py -pytest.ini

If you cd into the tests directory and run pytest from the cmd line the tests in the NotebookTest.py file should run. It will find both notebooks in the notebooks folder and test them (using the parameterize feature of pytest). I think you'll find that the first one fails because the random image in the saved notebook is not the same as the one that gets generated when the notebook runs. I expect this...however, in the pytest.ini file I have told pytest-notebook to ignore a bunch of stuff including images md5s that dont' match.

The second notebook will pass. I took the same commands from the pytest.ini file and inserted them into the metadata of the notebook, near the bottom.

So, it seems that the plug-in ignores the info in pytest.ini file.

It's either a bug, or I'm doing something wrong. Either way, I really appreciate your help.

[tuanng-cognite]

@chrisjsewell any update on this?

[jgostick]

there is an alternative pytest plugin that we used, called nbval.