testMultipleThreads failed

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that our python-snimpy 1.0.1 package can get stuck during tests or fail like this (in a 1-core VM):

=================================== FAILURES ===================================
________________________ TestSnmp3.testMultipleThreads _________________________

self = <test_snmp.TestSnmp3 testMethod=testMultipleThreads>

    @unittest.skipIf(platform.python_implementation() == "PyPy",
                     "unreliable test with Pypy")
    def testMultipleThreads(self):
        """Test with multiple sessions in different threads."""
        count = 20
        agents = []
        for i in range(count):
            agents.append(self.addAgent('community{}'.format(i),
                                        'community{}-authpass'.format(i),
                                        'community{}-privpass'.format(i)))
        ooid = mib.get('SNMPv2-MIB', 'sysDescr').oid + (0,)

        threads = []
        successes = []
        failures = []
        lock = multiprocessing.Lock()

        # Start one thread
        def run(i):
            params = self.setUpSession(agents[i], 'community{}'.format(i))
            session = snmp.Session(**params)
            session.timeout = 10 * 1000 * 1000
            oid, a = session.get(ooid)[0]
            exp = ("Snimpy Test Agent community{}".format(i)).encode('ascii')
            with lock:
                if oid == ooid and \
                   a == exp:
                    successes.append("community{}".format(i))
                else:
                    failures.append("community{}".format(i))
        for i in range(count):
            threads.append(threading.Thread(target=run, args=(i,)))
        for i in range(count):
            threads[i].start()
        for i in range(count):
            threads[i].join()
        self.assertEqual(failures, [])
>       self.assertEqual(sorted(successes),
                         sorted(["community{}".format(i)
                                 for i in range(count)]))
E       AssertionError: Lists differ: ['com[16 chars]nity10', 'community11', 'community12', 'commun[204 chars]ty9'] != ['com[16 chars]nity1', 'community10', 'community11', 'communi[218 chars]ty9']
E
E       First differing element 1:
E       'community10'
E       'community1'
E
E       Second list contains 1 additional elements.
E       First extra element 19:
E       'community9'
E
E         ['community0',
E       +  'community1',
E          'community10',
E          'community11',
E          'community12',
E          'community13',
E          'community14',
E          'community15',
E          'community16',
E          'community17',
E          'community18',
E          'community19',
E          'community2',
E          'community3',
E          'community4',
E          'community5',
E          'community6',
E          'community7',
E          'community8',
E          'community9']

tests/test_snmp.py:353: AssertionError
=============================== warnings summary ===============================
tests/test_snmp.py::TestSnmp3::testMultipleThreads
  /usr/lib/python3.9/site-packages/_pytest/threadexception.py:73: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-42

  Traceback (most recent call last):
    File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
      self.run()
    File "/usr/lib64/python3.9/threading.py", line 917, in run
      self._target(*self._args, **self._kwargs)
    File "/home/abuild/rpmbuild/BUILD/snimpy-1.0.1/tests/test_snmp.py", line 338, in run
      oid, a = session.get(ooid)[0]
    File "/home/abuild/rpmbuild/BUILDROOT/python-snimpy-1.0.1-0.0.x86_64/usr/lib64/python3.9/site-packages/snimpy/snmp.py", line 296, in get
      return self._op(self._cmdgen.getCmd, *oids)
    File "/home/abuild/rpmbuild/BUILDROOT/python-snimpy-1.0.1-0.0.x86_64/usr/lib64/python3.9/site-packages/snimpy/snmp.py", line 268, in _op
      raise SNMPException(str(errorIndication))
  snimpy.snmp.SNMPException: No SNMP response received before timeout

    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_snmp.py::TestSnmp3::testMultipleThreads - AssertionError: L...
======== 1 failed, 265 passed, 1 xfailed, 1 warning in 89.60s (0:01:29) ========

[vincentbernat]

I don't think that running tests on a 1-core VM is a good idea as it may trigger many "bugs" like this (we had the same debate in Debian). It triggers bugs in the test suite that are not real bugs and this takes too much time to fix these. What I could do is skip the test when we detect a single core.

[bmwiedemann]

Skipping is fine as long as you are sure it does not hide unreliable code behavior.