warp
warp copied to clipboard
[BUG] NaN in test_tile_cholesky_cholesky_multiple_rhs_cpu on Jetson Thor
Bug Description
The test reports a "nan location mismatch" between the Z_wp and Z_np matrices.
It only appears to happen when running the full test suite. Not when running the test in isolation.
System Information
No response
@daedalus5 ran into it during a merge request today, so the failure is rare but present on our CI/CD
======================================================================
test_tile_cholesky_cholesky_multiple_rhs_cpu (warp.tests.tile.test_tile_cholesky.TestTileCholesky.test_tile_cholesky_cholesky_multiple_rhs_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builds/omniverse/warp/warp/tests/unittest_utils.py", line 258, in test_func
func(self, device, **kwargs)
File "/builds/omniverse/warp/warp/tests/tile/test_tile_cholesky.py", line 147, in test_tile_cholesky_cholesky_multiple_rhs
np.testing.assert_allclose(Z_wp.numpy(), Z_np)
File "/builds/omniverse/warp/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1718, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/builds/omniverse/warp/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 818, in assert_array_compare
flagged = func_assert_same_pos(x, y, func=isnan, hasval='nan')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/builds/omniverse/warp/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 787, in func_assert_same_pos
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0
nan location mismatch:
ACTUAL: array([[ 5.304688, 11.429687, 17.554688, 23.679688, 29.804688, 35.929688,
42.054687, 48.179688],
[ 5.523437, 12.148437, 18.773438, 25.398437, 32.023438, nan,...
DESIRED: array([[ 5.304687, 11.429687, 17.554688, 23.679688, 29.804688, 35.929688,
42.054688, 48.179688],
[ 5.523437, 12.148438, 18.773437, 25.398437, 32.023437, 38.648437,...
----------------------------------------------------------------------