alibi
alibi copied to clipboard
Address drop in testing coverage due to excluding test files
After excluding test files themselves from test coverage reporting in #751 it has become clear that our actual coverage is smaller than expected. This task is to first list the areas where our testing can be improved and second to start addressing those.
Link to coverage report on master
: https://app.codecov.io/gh/SeldonIO/alibi
Note: it may be useful to use code exclusion where appropriate, see here. The coverage
tool also supports excluding certain line patterns via a configuration file.
Note on disambiguation between various tools
- Coverage reports are produced by the
coverage
tool, it's config file is managed in.coveragerc
(currently doesn't exist) - The coverage reports are automatically produced by
pytest
using thepytest-cov
plugin (which callscoverage
under the hood) and appropriate command-line flags. - The
codecov
tool takes these reports and uploads them to codecov.com for inspection and detailed analysis. This is done at the end of a CI run. The tool can also manage conditions, e.g. for when PRs are approved based on coverage, for which branches to upload reports etc. The configuration is in thecodecov.yml
file.
Note on testing plotting functions A lot of the untested code is plotting functions for which we should make a decision going forward. Some options include
- No testing and excluding from coverage the relevant functions
- Some integration tests of the public plotting functions which should cover a lot of the private plotting functions, but perhaps not all logical branches. This should be reasonably straightforward to accomplish.
- More granular testing...
Some useful tips on testing plotting code can be found here: https://stackoverflow.com/questions/27948126/how-can-i-write-unit-tests-against-code-that-uses-matplotlib. @ascillitoe @mauicv