robotology
Flaky Python pybind11 tests on macOS
Sometimes (around 1 in 10 runs, as an order of magnitude) the macos pybind11_idyntree_test fail with the following message:
65/65 Test #65: pybind11_idyntree_test ................................***Failed 0.37 sec
Errors while running CTest
...................................................................................E./Users/runner/miniconda3/envs/test/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpxis1j9yn'>
_warnings.warn(warn_message, ResourceWarning)
./Users/runner/miniconda3/envs/test/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp5wh0fys8'>
_warnings.warn(warn_message, ResourceWarning)
............
======================================================================
ERROR: test_exporter (test_idyntree_modelio_urdf.IDynTreeModelIoUrdfTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/runner/work/idyntree/idyntree/bindings/pybind11/tests/test_idyntree_modelio_urdf.py", line 30, in test_exporter
self.assertTrue(exporter.export_model_to_string())
ValueError: Failed to export model.
----------------------------------------------------------------------
Ran 98 tests in 0.018s
FAILED (errors=1)
98% tests passed, 1 tests failed out of 65
Total Test time (real) = 21.20 sec
The following tests FAILED:
65 - pybind11_idyntree_test (Failed)
Example of this failure:
- https://github.com/robotology/idyntree/runs/8148765522
- https://github.com/robotology/idyntree/runs/8089965230
I am not sure if the warning ResourceWarning: Implicitly cleaning up <TemporaryDirector is related to this, and if we can do anything to avoid this problem.
fyi @GiulioRomualdi @francesco-romano
ValueError: Failed to export model.
Probably this is forwarded from the C++. Can we get more details on why the model failed to export? (It might be because we have a bool returning there?)
I do not think the resource warning is the problem. AFAIK resources are cleaned up automatically when the object goes out of scope. But you are right that it is strange to have a warning for something that is WAI. Maybe we should check better
ValueError: Failed to export model.
Probably this is forwarded from the C++. Can we get more details on why the model failed to export? (It might be because we have a bool returning there?)
I do not think the resource warning is the problem. AFAIK resources are cleaned up automatically when the object goes out of scope. But you are right that it is strange to have a warning for something that is WAI. Maybe we should check better
On 3 Sep 2022, at 12:28, Silvio Traversaro @.***> wrote:
https://github.com/robotology/idyntree/runs/8166548692 https://github.com/robotology/idyntree/runs/8166548692 — Reply to this email directly, view it on GitHub https://github.com/robotology/idyntree/issues/1027#issuecomment-1236091608, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWO7B5RHU22AFD3W2DC6TV4MR47ANCNFSM6AAAAAAQDADZTY. You are receiving this because you were mentioned.
Nice. I originally replied from email but the message never got to GitHub. This is why I replied again directly in the app. Only one month of delay.. not bad :D
It happened again today:
67/67 Test #67: pybind11_idyntree_test ................................***Failed 0.21 sec
...................................................................................E./Users/runner/micromamba/envs/idyntreedev/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/gh/0cww6vn93nv_jjsw3xkm0b4m0000gn/T/tmp_gp2lzw9'>
_warnings.warn(warn_message, ResourceWarning)
./Users/runner/micromamba/envs/idyntreedev/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/gh/0cww6vn93nv_jjsw3xkm0b4m0000gn/T/tmpuaoc6o0j'>
_warnings.warn(warn_message, ResourceWarning)
............
======================================================================
ERROR: test_exporter (test_idyntree_modelio_urdf.IDynTreeModelIoUrdfTest.test_exporter)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/runner/work/idyntree/idyntree/bindings/pybind11/tests/test_idyntree_modelio_urdf.py", line 30, in test_exporter
self.assertTrue(exporter.export_model_to_string())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Failed to export model.
----------------------------------------------------------------------
Ran 98 tests in 0.012s
FAILED (errors=1)
Errors while running CTest
https://github.com/robotology/idyntree/actions/runs/7982942399/job/21797250676
I think I, too, might have observed some flakiness in some iDynTree python function