Comparing output mime bundles

[SylvainCorlay]

Running the following code in a cell

from IPython.display import Image
Image('some_image.png')

then replacing the content with

from IPython.display import Image
Image('another_image.png')

and saving the notebook without executing shoud result in an inconsistent state.

However, nbval does not appear to see the difference.

[takluyver]

IIRC, differences in images are ignored unless you're using the nbdime integration, because when you're creating a plot or something, even running the exact same code often doesn't produce a byte-for-byte identical result, and a failure saying that two huge blobs of base64 don't match isn't very helpful. Obviously when you're loading a file, it should be byte-for-byte identical, though.

Maybe if you use --nbval-lax, then cells specifically marked for checking should have all their outputs checked. Or maybe we should have a separate marker to say 'check even the images on this'.

[SylvainCorlay]

OK, I was surprised because differences in mpl figures are picked up.

[vidartf]

I agree, that the content of this set should be more easily configurable. Input from users (@SylvainCorlay) on how to best configure this would be helpful for getting it right :)