Add tests to pysteps modules

[aperezhortal]

Now, the pySTEPS tests are mostly done by running the examples. It is a good idea to implement scripts that test different functions. A good and simple testing framework is pytest.

By doing so, we can test the library after any change that we want to commit.

One of the best advantages of implementing these tests is that we can set continuous integration service used to build and test projects hosted at GitHub, like Travis-CI. This is used for example in py-art. Many of the CI servers support integration with github. By doing so, after each commit, the tests can be run under different environments and the results are visible in the commits tab in github (see pyart green checks for example ). Also, the pull request can be automatically tested before merging.

I created a branch with a script to test the interfaces as an example: https://github.com/aperezhortal/pysteps/tree/tests/test

To run this tests, execute pytest in the test folder (pytest package is needed).

The output looks like this:

============================= test session starts ============================== platform linux -- Python 3.6.7, pytest-4.0.1, py-1.7.0, pluggy-0.8.0

test_interfaces.py . [100%]

=========================== 1 passed in 1.25 seconds =========================== Process finished with exit code 0

[wcwoo]

pytest is a really useful tool to validate codes. While not familiar with Travis-CI, I found GitLab CI + pytest necessary for me to identify issues early.

I just ran pytest but got 2 errors. Is my new installation good for use?

(pysteps) swirls@swirls-VirtualBox:~/pysteps$ pytest --pyargs pysteps
======================================================================================== test session starts =========================================================================================
platform linux -- Python 3.6.7, pytest-4.2.1, py-1.7.0, pluggy-0.8.1
rootdir: /home/swirls/pysteps, inifile:
collected 7 items                                                                                                                                                                                    

tests/test_interfaces.py .F...F.                                                                                                                                                               [100%]

============================================================================================== FAILURES ==============================================================================================
____________________________________________________________________________________ test_extrapolation_interface ____________________________________________________________________________________

    def test_extrapolation_interface():
        """ Test the extrapolation module interface"""
    
        from pysteps import extrapolation
        from pysteps.extrapolation import semilagrangian
    
        method_getter = extrapolation.interface.get_method
    
        valid_names_func_pair = [('semilagrangian', semilagrangian.extrapolate)]
        invalid_names = ['euler', 'LAGRANGIAN']
>       _generic_interface_test(method_getter, valid_names_func_pair, invalid_names)

../.local/lib/python3.6/site-packages/pysteps-0.2-py3.6-linux-x86_64.egg/pysteps/tests/test_interfaces.py:49: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

method_getter = <function get_method at 0x7fce4d9776a8>, valid_names_func_pair = [('semilagrangian', <function extrapolate at 0x7fce4d9e6510>)], invalid_names = ['euler', 'LAGRANGIAN']

    def _generic_interface_test(method_getter,
                                valid_names_func_pair,
                                invalid_names):
        for name, expected_function in valid_names_func_pair:
            error_message = "Error getting '{}' function.".format(name)
>           assert method_getter(name) == expected_function, error_message
E           AssertionError: Error getting 'semilagrangian' function.
E           assert (<function initialize at 0x7fce4d9e6598>, <function extrapolate at 0x7fce4d9e6510>) == <function extrapolate at 0x7fce4d9e6510>
E            +  where (<function initialize at 0x7fce4d9e6598>, <function extrapolate at 0x7fce4d9e6510>) = <function get_method at 0x7fce4d9776a8>('semilagrangian')

../.local/lib/python3.6/site-packages/pysteps-0.2-py3.6-linux-x86_64.egg/pysteps/tests/test_interfaces.py:15: AssertionError
______________________________________________________________________________________ test_nowcasts_interface _______________________________________________________________________________________

    def test_nowcasts_interface():
        """ Test the nowcasts module interface"""
    
        from pysteps.nowcasts import steps
        from pysteps.nowcasts import extrapolation
        method_getter = pysteps.nowcasts.interface.get_method
    
        valid_names_func_pair = [('extrapolation', extrapolation.forecast),
                                 ('steps', steps.forecast)]
    
        invalid_names = ['extrap', 'step']
        _generic_interface_test(method_getter, valid_names_func_pair, invalid_names)
    
        # Test eulerian persistence method
        precip = numpy.random.rand(100, 100)
        velocity = numpy.random.rand(100, 100)
        num_timesteps = 10
        for name in ["eulerian", "EULERIAN"]:
>           forecast = method_getter(name)(precip, velocity, num_timesteps)
E           TypeError: eulerian_persistence() missing 1 required positional argument: 'num_timesteps'

../.local/lib/python3.6/site-packages/pysteps-0.2-py3.6-linux-x86_64.egg/pysteps/tests/test_interfaces.py:186: TypeError
================================================================================= 2 failed, 5 passed in 0.70 seconds =================================================================================

[aperezhortal]

Hi @wcwoo. Thanks for pointing this out. This is not a problem with the installation itself. It is an issue with the same eulerian_persistence function that is being used in two different interfaces (change made in commit 7e0b82d6789e2c864957b9a67ca09dfbc2565e7e). It is now solved (commit f8e4d3c81f324c8c355f210d45275c3132c8d44f).

[cvelascof]

Hi @aperezhortal,

I have started to work in a Travis CI configuration for pysteps. See https://github.com/cvelascof/pysteps/blob/master/.travis.yml

So far, this configuration https://travis-ci.org/cvelascof/pysteps/jobs/502472265/config is able to install conda, all packages for pysteps, and pysteps itself.

However, when I try to import pysteps before to run any tests I got the following error message:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/travis/build/cvelascof/pysteps/pysteps/__init__.py", line 15, in <module>
    from . import motion
  File "/home/travis/build/cvelascof/pysteps/pysteps/motion/__init__.py", line 3, in <module>
    from .interface import get_method
  File "/home/travis/build/cvelascof/pysteps/pysteps/motion/interface.py", line 5, in <module>
    from pysteps.motion.vet import vet
  File "/home/travis/build/cvelascof/pysteps/pysteps/motion/vet.py", line 30, in <module>
    from pysteps.motion._vet import _warp, _cost_function
ModuleNotFoundError: No module named 'pysteps.motion._vet'

https://travis-ci.org/cvelascof/pysteps/jobs/502472265#L1488

I guess that there is something incorrect in the way that cython compiles vet but I cannot find what it is wrong. Any ideas?

[wcwoo]

Maybe you need to make the module searchable from your python by, say, exporting PYTHONPATH=...?

[dnerini]

[...]

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/travis/build/cvelascof/pysteps/pysteps/__init__.py", line 15, in <module>
    from . import motion
  File "/home/travis/build/cvelascof/pysteps/pysteps/motion/__init__.py", line 3, in <module>
    from .interface import get_method
  File "/home/travis/build/cvelascof/pysteps/pysteps/motion/interface.py", line 5, in <module>
    from pysteps.motion.vet import vet
  File "/home/travis/build/cvelascof/pysteps/pysteps/motion/vet.py", line 30, in <module>
    from pysteps.motion._vet import _warp, _cost_function
ModuleNotFoundError: No module named 'pysteps.motion._vet'

https://travis-ci.org/cvelascof/pysteps/jobs/502472265#L1488

I guess that there is something incorrect in the way that cython compiles vet but I cannot find what it is wrong. Any ideas?

This seems related to the problem discussed in Issue #40.

[wcwoo]

I forgot that I raised that issue. :)

[cvelascof]

Yes. Thanks. That did the trick. Running the tests from ~ fixed it.

https://travis-ci.org/cvelascof/pysteps/builds/502524712

I will do a merge request soon.

[cvelascof]

@aperezhortal, @dnerini have you tried to run the test_motion outside of Travis? I mean in your own computer. In my case, test_motion is taking so much time to run. It passes but in comparison with the other tests is sooooo slow. I recall that it used to be as fast as the other tests. Any ideas why it change ?

[dnerini]

Yes, I run the tests locally on my computer and it is true that test_motion takes some time, mostly because it has to compute multiple times VET and make sure that it converges to the right solution (by increasing the number of iterations).

But are you sure that is used to be faster?

Perhaps one way to speed up the test could be to reduce the dimension of the domain (e.g. upscaling the reference field).

[aperezhortal]

@cvelascof, that test indeed takes some time. I will test upscaling the reference fields as @dnerini suggested to speed up the tests. Hopefully, it will be straightforward.

[cvelascof]

@aperezhortal thanks. that would be nice to have. Checking new uni-tests is taking too long now.

[aperezhortal]

@cvelascof @dnerini, I tried upscaling the domain by 2, but the quality of the VET field was much lower. It converges to a local minimum... To speed up the tests, I only left only one VET test and commented out the others.

This is a temporary patch, while I look for a way to optimize the _vet module. ( 24e4f8a )

[dnerini]

Good news: we have (finally) achieved more than 50% coverage!

[dnerini]

Let's set a 90% coverage goal for pysteps V2.