message_ix icon indicating copy to clipboard operation
message_ix copied to clipboard
verified publisher icon iiasa

Make more tests robust & independent

Open glatterf42 opened this issue 1 year ago • 8 comments

Unfortunately, even though #784 improved the tests a lot, we still seem to see flaky behaviour. So let's continue the same kind of work from #784 on these tests:

  • [ ] 
    FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_add_share_mode_lo - assert False
 +  where False = <function isclose at 0x111d84700>(0.5416666666666666, 0.525)
 +    where <function isclose at 0x111d84700> = np.isclose

glatterf42 avatar Mar 11 '24 07:03 glatterf42

It seems like this flaky behaviour wasn't completely resolved by the first PR:

  • [ ] 
    ERROR message_ix/tests/tools/test_add_year.py::test_add_year - RuntimeError: unhandled Java exception: Error opening GDX file: Expected data marker (BOI) not found in GDX file
On: windows-latest-py3.8

glatterf42 avatar Mar 13 '24 06:03 glatterf42

In addition, I'm seeing another error for a test that we thought we made non-flaky:

  • [ ] 
    ERROR message_ix/tests/test_feature_bound_activity_shares.py::test_b_a_u_all_modes[299-5] - FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/work/message_ix/message_ix/message_ix/model/225a'
On: macos-latest-py3.8, macos-latest-py3.9, macos-latest-py3.10, ubuntu-latest-py3.9, ubuntu-latest-py3.11

This is happening for macos-latest-py3.8, macos-latest-py3.9, and macos-latest-py3.10, but for 3.9 and 3.10, the test case in question is rather [299-2] (and everything else is exactly the same). Edit: now this is also happening with [299-3] on macos-latest-py3.9 and [299-1] on macos-latest-py3.10.

glatterf42 avatar Mar 13 '24 11:03 glatterf42

There's a new flaky error for test_add_year:

  • [ ] 
    ERROR message_ix/tests/tools/test_add_year.py::test_add_year_cli - RuntimeError: unhandled Java exception: Error opening GDX file: Expected data marker (SYMBOL_2) not found in GDX file
On: windows-latest-py3.10

glatterf42 avatar Mar 22 '24 06:03 glatterf42

Another one bites the dust:

  • [ ] 
    FAILED message_ix/tests/test_cli.py::test_copy_model - AssertionError: assert 1 == 0
 +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code
On: macos-latest-py3.9, macos-latest-py3.11

glatterf42 avatar Mar 29 '24 07:03 glatterf42

Another flaky test appeared today:

  • [ ] 
     FAILED message_ix/tests/test_util.py::test_testing_make_scenario - RuntimeError: unhandled Java exception: The index set 'emission' does not have an element 'CO2'!
On: macos-13-py3.10

glatterf42 avatar May 07 '24 06:05 glatterf42

There goes the next one:

  • [ ] 
    FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_add_share_mode_lo - assert False
 +  where False = <function isclose at 0x000001BF5AF800F0>(0.5416666666666666, 0.525)
 +    where <function isclose at 0x000001BF5AF800F0> = np.isclose
On: windows-latest-py3.11

glatterf42 avatar May 22 '24 13:05 glatterf42

Postponing as internal PRs are not that time-sensitive anyway.

glatterf42 avatar May 23 '24 12:05 glatterf42

  • [ ] 
     FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_commodity_share_lo - assert False
 +  where False = <function isclose at 0x10fe1d8b0>(0.0, 0.15384615384615385)
 +    where <function isclose at 0x10fe1d8b0> = np.isclose
 +    and   0.15384615384615385 = <function test_commodity_share_lo.<locals>.calc_share at 0x161a71ca0>(<message_ix.core.Scenario object at 0x158b22370>)
On: macos-latest-py3.8

glatterf42 avatar Jun 03 '24 05:06 glatterf42