nbdev icon indicating copy to clipboard operation
nbdev copied to clipboard

How to guide for testing

Open hamelsmu opened this issue 2 years ago • 14 comments

A guide that shows how we test notebooks:

  • fastcore.test
  • How to write unit tests in notebooks
  • test flags
  • How to test nbdev_test, and how to interpret the outputs esp if there is a failure
  • Recommendations for style
  • How this may be different than pytest etc

hamelsmu avatar Aug 26 '22 02:08 hamelsmu

This would be really cool! Currently, nbdev_test does not produce any interpretable output, only lists the failing notebooks without the error message (like nbdev1 used to).

tcapelle avatar Sep 02 '22 08:09 tcapelle

@tcapelle it’s supposed to print each error and corresponding cell source. I looked into the nb_helpers repo, in case that’s what you’re referring to, and the errors aren’t printed there because the RichLogger class reconfigures the root logger with level=logging.ERROR, and nbdev_test use warning which logs at level WARNING which is lower than ERROR

cc @hamelsmu @jph00 - maybe we want to change the way we print those errors?

seeM avatar Sep 02 '22 08:09 seeM

@seeM I thought maybe you set the logging level to error or were in the process of doing that? Or is that for something else?

hamelsmu avatar Sep 02 '22 21:09 hamelsmu

@tcapelle could you try again with latest master and see if nbdev_test produces interpretable output for you?

seeM avatar Sep 08 '22 09:09 seeM

@hamelsmu do you think this is sufficiently covered in https://nbdev.fast.ai/tutorials/best_practices.html?

seeM avatar Sep 15 '22 02:09 seeM

No I don't think so, I think we can talk a lot more about tests esp:

  • Difference from pytest (ex: why you don't need fixtures)
  • test flags
  • When to incorporate tests into visible documentation and when not to

hamelsmu avatar Sep 15 '22 03:09 hamelsmu

Ohh missed this! thanks guys, and I actually figured this out :P

tcapelle avatar Oct 04 '22 16:10 tcapelle

Hi! I'm very interested in developing a plotting package using nbdev. However, most graphical testing plugins are for IDE development. Do you have any suggestion on graphical testing in notebooks?

Jacobluke- avatar Nov 14 '22 09:11 Jacobluke-

Sounds interesting @Jacobluke-! I'm not familiar with "graphical testing", could you please share some graphical testing plugins you've enjoyed using?

seeM avatar Nov 15 '22 13:11 seeM

Hi @seeM , thanks for your reply.

Basically "graphical testing" ensures that our package continue to work in a consistent way. The plugin we are using is pytest-mpl, and here is a brief example on how to using this plugin.

Jacobluke- avatar Nov 15 '22 13:11 Jacobluke-

One pitfall I've run into that may be worth addressing in such a guide: when I forget to export functions (or imports), tests will still pass because the non-exported stuff is loaded in the notebook. I guess putting some tests in a separate notebook would guard against this?

hugetim avatar Feb 05 '23 04:02 hugetim

Message ID: @.***>I'm not sure I fully understand your question/concern. If you run nbdev_prepare or nbdev_test it should test correctly, shouldn't it?

jph00 avatar Feb 05 '23 05:02 jph00

Message ID: @.***>Oh sorry I understand what you're saying now. Yes that can be an issue, although if you're exporting something that you never actually use anywhere else, then hopefully it's not a big problem! ;)

jph00 avatar Feb 05 '23 05:02 jph00

Right, it is often caught in a test in a separate notebook if it is used elsewhere, but I won't always have exhaustive test coverage elsewhere if I think its tested in the source notebook. To be more specific about the way this has bit me, it's usually a private helper function or global variable defined in the module scope that I forget to #|export, not something that would be directly called from elsewhere.

hugetim avatar Feb 05 '23 17:02 hugetim