warp icon indicating copy to clipboard operation
warp copied to clipboard

[BUG] NaN in test_tile_cholesky_cholesky_multiple_rhs_cpu on Jetson Thor

Open c0d1f1ed opened this issue 3 months ago • 1 comments

Bug Description

The test reports a "nan location mismatch" between the Z_wp and Z_np matrices.

It only appears to happen when running the full test suite. Not when running the test in isolation.

System Information

No response

c0d1f1ed avatar Oct 17 '25 01:10 c0d1f1ed

@daedalus5 ran into it during a merge request today, so the failure is rare but present on our CI/CD

======================================================================
test_tile_cholesky_cholesky_multiple_rhs_cpu (warp.tests.tile.test_tile_cholesky.TestTileCholesky.test_tile_cholesky_cholesky_multiple_rhs_cpu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/omniverse/warp/warp/tests/unittest_utils.py", line 258, in test_func
    func(self, device, **kwargs)
  File "/builds/omniverse/warp/warp/tests/tile/test_tile_cholesky.py", line 147, in test_tile_cholesky_cholesky_multiple_rhs
    np.testing.assert_allclose(Z_wp.numpy(), Z_np)
  File "/builds/omniverse/warp/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1718, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/builds/omniverse/warp/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 818, in assert_array_compare
    flagged = func_assert_same_pos(x, y, func=isnan, hasval='nan')
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/omniverse/warp/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 787, in func_assert_same_pos
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0
nan location mismatch:
 ACTUAL: array([[ 5.304688, 11.429687, 17.554688, 23.679688, 29.804688, 35.929688,
        42.054687, 48.179688],
       [ 5.523437, 12.148437, 18.773438, 25.398437, 32.023438,       nan,...
 DESIRED: array([[ 5.304687, 11.429687, 17.554688, 23.679688, 29.804688, 35.929688,
        42.054688, 48.179688],
       [ 5.523437, 12.148438, 18.773437, 25.398437, 32.023437, 38.648437,...
----------------------------------------------------------------------

shi-eric avatar Oct 29 '25 23:10 shi-eric