Adjust for JPype1 v1.4.1

[khaeru]

With the release of JPype1 version 1.4.1, CI jobs have started to fail, e.g. here, with messages like:

=================================== FAILURES ===================================
________________________ TestCachingBackend.test_del_ts ________________________

self = <ixmp.tests.backend.test_base.TestCachingBackend object at 0x7f06715dc0d0>
test_mp = <ixmp.core.platform.Platform object at 0x7f05ffaf7cd0>

    def test_del_ts(self, test_mp):
        """Test CachingBackend.del_ts()."""
        # Since CachingBackend is an abstract class, test it via JDBCBackend
        backend = test_mp._backend
        cache_size_pre = len(backend._cache)
    
        # Load data, thereby adding to the cache
        s = make_dantzig(test_mp)
        s.par("d")
    
        # Cache size has increased
        assert cache_size_pre + 1 == len(backend._cache)
    
        # Delete the object; associated cache is freed
        del s
    
        # Objects were invalidated/removed from cache
>       assert cache_size_pre == len(backend._cache)
E       AssertionError: assert 0 == 1
E        +  where 1 = len({(139663741246368, 'par', 'd'):            i         j  value unit\n0    seattle  new-york    2.5   km\n1    seattle   chicago    1.7   km\n2    seattle    topeka    1.8   km\n3  san-diego  new-york    2.5   km\n4  san-diego   chicago    1.8   km\n5  san-diego    topeka    1.4   km})
E        +    where {(139663741246368, 'par', 'd'):            i         j  value unit\n0    seattle  new-york    2.5   km\n1    seattle   chicago    1.7   km\n2    seattle    topeka    1.8   km\n3  san-diego  new-york    2.5   km\n4  san-diego   chicago    1.8   km\n5  san-diego    topeka    1.4   km} = <ixmp.backend.jdbc.JDBCBackend object at 0x7f05ffaf6890>._cache

ixmp/tests/backend/test_base.py:148: AssertionError
_________________________________ test_del_ts __________________________________

    def test_del_ts():
        mp = ixmp.Platform(
            backend="jdbc",
            driver="hsqldb",
            url="jdbc:hsqldb:mem:test_del_ts",
        )
    
        # Number of Java objects referenced by the JDBCBackend
        N_obj = len(mp._backend.jindex)
    
        # Create a list of some Scenario objects
        N = 8
        scenarios = [make_dantzig(mp)]
        for i in range(1, N):
            scenarios.append(scenarios[0].clone(scenario=f"clone {i}"))
    
        # Number of referenced objects has increased by 8
        assert len(mp._backend.jindex) == N_obj + N
    
        # Pop and free the objects
        for i in range(N):
            s = scenarios.pop(0)
    
            message = "\n".join(map(str, gc.get_referrers(s)))
    
            # The variable 's' is the only reference to this Scenario object
>           assert 1 == getrefcount(s) - 1, (message, gc.garbage)
E           AssertionError: ("<frame at 0x7f05ffa8e700, file '/home/runner/work/ixmp/ixmp/ixmp/backend/jdbc.py', line 694, code check_out>
E             <frame...
E             <frame at 0x7f05ffa14b80, file '/home/runner/work/ixmp/ixmp/ixmp/testing/data.py', line 215, code make_dantzig>", [])
E           assert 1 == (19 - 1)
E            +  where 19 = getrefcount(<ixmp.core.scenario.Scenario object at 0x7f05ffaabb20>)

ixmp/tests/backend/test_jdbc.py:338: AssertionError

• Essentially, some change in this JPype release has caused the Python garbage collection mechanisms, on which ixmp.backend.CachingBackend relies, to misfire.
  • At the end of various method calls, function-local references to the Scenario/TimeSeries object are not released/GC'd, so that when (for example) the del s line is reached, these persisting references / non-zero refcount prevent the object itself, and associated cached item values, from being garbage-collected.
• The issue did not occur with JPype 1.4.0.
• See the diff between JPype1 1.4.0 and 1.4.1. The changes in the file pyjp_value.cpp appear relevant.
• In #458 (commits) I encountered similar test failures and made changes such as 0db14db75928cf64ae66c2b5be4079ada5f5dc8d to adjust.
  • Those errors began to appear with the release of pandas 1.5.0, but did not occur with pandas 1.4.4; in both cases with JPype1 1.4.0. See here.
  • In other words, a change in pandas with no change in JPype1 caused the errors to begin to appear.
  • From this, I guess this may have something to do with C/C++-level handling of memory and references; the way these are overridden or adjusted by C/C++ code in pandas and in JPype; and perhaps interactions of the two.

To mitigate

• Avoid using ixmp with JPype1 1.4.1 if the CachingBackend behaviour is needed.
• [x] Adjust test suite to exclude this version → #464

To fix

• [ ] Inquire upstream with JPype; maybe produce a minimum working example to reproduce; maybe wait to see if others file about a similar root issue and a fix is issued.
• [ ] Make any necessary adjustments in ixmp per se; then remove the above exclusion.