FunFact
FunFact copied to clipboard
yhtang
JOSS review: unit tests
This is related to the review of FunFact for JOSS (see https://github.com/openjournals/joss-reviews/issues/4502)
I ran the unit tests (on the branch tags/1.0 which I hope is the correct one) and this resulted in a warning due to log(zero/negative) and 12 failing tests, mostly with the error
NotImplementedError: NumPy backend does not support AutoGrad.
Is this the intended behaviour or why are there failing unit tests? Thank you a lot!
Below you find my test results, also including a coverage report for the sake of completeness:
funfact/test_loss.py::test_vectorization[KLDivergence]
/home/schaipp/private/FunFact/funfact/loss.py:101: RuntimeWarning: invalid value encountered in log
return ab.multiply(target, ab.log(ab.divide(target, model)))
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.9.12-final-0 -----------
Name Stmts Miss Cover
--------------------------------------------------------------------------
funfact/__init__.py 18 0 100%
funfact/algorithm.py 73 63 14%
funfact/backend/__init__.py 2 0 100%
funfact/backend/_context.py 8 0 100%
funfact/backend/_jax.py 48 41 15%
funfact/backend/_meta.py 2 2 0%
funfact/backend/_numpy.py 25 4 84%
funfact/backend/_proxy.py 51 2 96%
funfact/backend/_torch.py 77 74 4%
funfact/backend/test_proxy.py 46 0 100%
funfact/conditions.py 55 4 93%
funfact/context.py 16 2 88%
funfact/initializers.py 67 0 100%
funfact/lang/__init__.py 16 6 62%
funfact/lang/_ast.py 57 0 100%
funfact/lang/_constants.py 4 0 100%
funfact/lang/_math.py 23 0 100%
funfact/lang/_semiring.py 8 0 100%
funfact/lang/_special.py 11 0 100%
funfact/lang/_terminal.py 125 3 98%
funfact/lang/_tplex.py 24 12 50%
funfact/lang/_tsrex.py 199 0 100%
funfact/lang/interpreter/__init__.py 12 0 100%
funfact/lang/interpreter/_ascii.py 48 0 100%
funfact/lang/interpreter/_base.py 187 29 84%
funfact/lang/interpreter/_einop.py 11 0 100%
funfact/lang/interpreter/_einop_compiler.py 83 8 90%
funfact/lang/interpreter/_elementwise.py 13 7 46%
funfact/lang/interpreter/_evaluation.py 31 14 55%
funfact/lang/interpreter/_indexness.py 48 7 85%
funfact/lang/interpreter/_initialization.py 50 8 84%
funfact/lang/interpreter/_latex.py 45 24 47%
funfact/lang/interpreter/_slicing_propagation.py 53 0 100%
funfact/lang/interpreter/_type_deduction.py 126 2 98%
funfact/lang/interpreter/_vectorize.py 48 9 81%
funfact/lang/interpreter/test_ascii.py 11 0 100%
funfact/lang/interpreter/test_base.py 56 2 96%
funfact/lang/interpreter/test_einop.py 45 0 100%
funfact/lang/interpreter/test_einop_compiler.py 13 0 100%
funfact/lang/interpreter/test_initialization.py 45 0 100%
funfact/lang/interpreter/test_slicing_propagation.py 49 0 100%
funfact/lang/interpreter/test_type_deduction.py 52 0 100%
funfact/lang/test_ast.py 40 2 95%
funfact/lang/test_semiring.py 12 0 100%
funfact/lang/test_special.py 15 0 100%
funfact/lang/test_terminal.py 93 0 100%
funfact/lang/test_tsrex.py 84 0 100%
funfact/loss.py 41 1 98%
funfact/model/__init__.py 2 0 100%
funfact/model/_factorization.py 87 1 99%
funfact/model/test_factorization.py 78 0 100%
funfact/optim.py 49 0 100%
funfact/test_algorithm.py 59 15 75%
funfact/test_conditions.py 18 0 100%
funfact/test_context.py 12 0 100%
funfact/test_initializers.py 69 0 100%
funfact/test_loss.py 74 0 100%
funfact/test_optim.py 84 2 98%
funfact/test_vectorization.py 87 0 100%
funfact/util/__init__.py 0 0 100%
funfact/util/iterable.py 36 0 100%
funfact/util/set.py 10 0 100%
funfact/util/test_iterable.py 36 0 100%
funfact/util/test_set.py 48 0 100%
funfact/vectorization.py 15 0 100%
--------------------------------------------------------------------------
TOTAL 3030 344 89%
========================== short test summary info ===========================
FAILED funfact/test_algorithm.py::test_simple[test_case0] - NotImplementedE...
FAILED funfact/test_algorithm.py::test_simple[test_case1] - NotImplementedE...
FAILED funfact/test_algorithm.py::test_simple[test_case2] - NotImplementedE...
FAILED funfact/test_algorithm.py::test_dtype - NotImplementedError: NumPy b...
FAILED funfact/test_algorithm.py::test_kwargs[True-first] - NotImplementedE...
FAILED funfact/test_algorithm.py::test_kwargs[True-2] - NotImplementedError...
FAILED funfact/test_algorithm.py::test_kwargs[True-None] - NotImplementedEr...
FAILED funfact/test_algorithm.py::test_kwargs[False-first] - NotImplemented...
FAILED funfact/test_algorithm.py::test_kwargs[False-2] - NotImplementedErro...
FAILED funfact/test_algorithm.py::test_kwargs[False-None] - NotImplementedE...
FAILED funfact/test_algorithm.py::test_returns - NotImplementedError: NumPy...
FAILED funfact/test_algorithm.py::test_penalty_weight - NotImplementedError...
================= 12 failed, 300 passed, 1 warning in 1.61s ==================
I think the reason for this is that I installed FunFact in an empty conda environment. So I have neither torch nor jax and then all autograd functions can of course not be used. Maybe this should be stated somewhere in the README? In the requirements.txt there is none of those packages specified.
Update on this: after installing torch the unit tests are succesfull. I would still recommend to make it more prominent in the README (and/or requirements.txt) that without having torch or jax installed, a major part of the functionalities of the package cannot be used.
Thanks for the feedback! We mentioned the installation of FunFact with specific autograd backends in the Installation page of our online docs. Perhaps there could be a better place for this information? What would be your suggestions?
Ah, thank you, indeed I did overlook this. Maybe put a link to this page on the Github README and the landing page of your documentation (that's where I usually look first if I would want to install your package)?