ert icon indicating copy to clipboard operation
ert copied to clipboard
verified publisher icon equinor

Task destroyed while pending

Open JHolba opened this issue 1 year ago • 3 comments

Describe the bug Getting errors from async when running scheduler tests

Task was destroyed but it is pending!
task: <Task cancelling name='job-0_task' coro=<Job.run() running at /home/jholba/equinor/ert/src/ert/scheduler/job.py:131> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[_done_callback() at /home/jholba/equinor/ert/src/_ert/async_utils.py:38]>
Task was destroyed but it is pending!
task: <Task cancelling name='job-1_task' coro=<Job.run() running at /home/jholba/equinor/ert/src/ert/scheduler/job.py:131> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[_done_callback() at /home/jholba/equinor/ert/src/_ert/async_utils.py:38]>

To reproduce pytest -n16 tests/integration_tests/scheduler could be flaky, but at least one error was reported on 3 different attempts

Expected behaviour no errors

Environment

  • OS: Fedora 39
  • ERT/Komodo release: git main
  • Python version: 3.11
  • Remote/HPC execution involved: no
  • unmanaged dev laptop

JHolba avatar Apr 25 '24 10:04 JHolba

Did it say which test this originated from? I am unable to reproduce it on macos.

jonathan-eq avatar Apr 26 '24 11:04 jonathan-eq

Running some in parallell, they fail when using a high count because it is running out of available ports.

FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[34-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[35-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[13-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[36-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[37-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[14-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[38-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[15-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[39-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[16-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[41-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[65-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[48-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[42-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[17-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[66-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[20-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[54-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[49-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[43-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[67-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[55-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[21-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[50-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[44-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[68-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[56-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[22-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[51-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[69-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[45-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[23-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[57-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[28-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[52-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[60-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[46-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[24-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[58-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[29-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[61-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[25-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[30-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[31-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[32-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[71-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[72-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[73-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[74-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[63-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
FAILED tests/integration_tests/scheduler/test_openpbs_driver.py::test_openpbs_driver_with_poly_example_failing_submit_fails_ert_and_propagates_exception_to_user[64-100] - ert.shared.port_handler.NoPortsInRangeException: No available ports in range range(51820, 51841).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 51 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================================================================================================================== 51 failed, 2323 passed, 11 warnings in 458.94s (0:07:38) ====================================================================================================================================

jonathan-eq avatar Apr 29 '24 06:04 jonathan-eq

it didn't say which test, no

JHolba avatar May 02 '24 08:05 JHolba

@xjules I am closing this as it has not been observed for a while.

jonathan-eq avatar May 15 '24 08:05 jonathan-eq