cctbx_project icon indicating copy to clipboard operation
cctbx_project copied to clipboard

libtbx testing broken

Open Anthchirp opened this issue 5 years ago • 0 comments

Test in #421 is failing, but libtbx testing mistakenly treats it as success and only notes the standard error output. The test is not listed in the "This-is-broken-this-is-fine" category.

libtbx.python "/Users/dlshudson/jenkins_slave/workspace/cctbx_mac_python3/build_dials/modules/cctbx_project/scitbx/suffixtree/test/tst_single.py" [OK] 0.2s
  Standard error:
(...)
    FAILED (errors=1)
libtbx.python "/Users/dlshudson/jenkins_slave/workspace/cctbx_mac_python3/build_dials/modules/cctbx_project/scitbx/glmtbx/tests/tst.py" [OK] 0.7s
libtbx.python "/Users/dlshudson/jenkins_slave/workspace/cctbx_mac_python3/build_dials/modules/cctbx_project/scitbx/iso_surface/tst_iso_surface.py" [OK] 9.2s
================================================================================

Tests finished. Elapsed time: 351.59s

(..)

Summary:
  Tests run                    : 423
  Failures                     : 0
  Warnings (possible failures) : 0
  Known Failures (  4)         : 4
  Known Unstable (  1)         : 0
  Stderr output (discouraged)  : 5

That is probably why this test failure was not picked up earlier.

I guess the reason for this is that this test seems to run unittest under the hood but the failure behaviour was not tested. I suggest converting the tests to pytest. That works in a predictable manner and doesn't swallow errors.

Anthchirp avatar Nov 26 '19 10:11 Anthchirp