MONAI icon indicating copy to clipboard operation
MONAI copied to clipboard

IntegrationClassification2D timeout

Open wyli opened this issue 3 years ago • 3 comments

Describe the bug

[2023-04-06T04:23:52.950Z] ======================================================================
[2023-04-06T04:23:52.950Z] ERROR: test_timing (__main__.IntegrationClassification2D)
[2023-04-06T04:23:52.950Z] ----------------------------------------------------------------------
[2023-04-06T04:23:52.950Z] Traceback (most recent call last):
[2023-04-06T04:23:52.950Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/tests/utils.py", line 633, in _wrapper
[2023-04-06T04:23:52.950Z]     raise timeout_error
[2023-04-06T04:23:52.950Z] multiprocessing.context.TimeoutError: 'test_timing' in '__main__' did not finish in 1000s.
[2023-04-06T04:23:52.950Z] 
[2023-04-06T04:23:52.950Z] ----------------------------------------------------------------------

(this happens randomly on A100 postmerge, works fine v100)

wyli avatar Apr 06 '23 07:04 wyli

seems to be addressed by limiting num_threads

export OMP_NUM_THREADS=8
export MKL_NUM_THREADS=8

wyli avatar Apr 06 '23 12:04 wyli

Still an issue

wyli avatar Apr 11 '23 06:04 wyli

@KumoLiu is testing on A100.

Thanks.

Nic-Ma avatar Apr 11 '23 06:04 Nic-Ma

no issue after the blossom migration

wyli avatar Aug 10 '23 10:08 wyli