RuntimeError: can't start new thread

[cmishra]

Hi,

Running 125*4 jobs I'm getting a slew of runtime errors with the following error message:

File "/speech7/cmishra11/library/dagobah/dagobah/core/core.py", line 429, in start self.tasks[task_name].start() File "/speech7/cmishra11/library/dagobah/dagobah/core/core.py", line 819, in start self._start_check_timer() File "/speech7/cmishra11/library/dagobah/dagobah/core/core.py", line 1018, in _start_check_timer self.timer.start() File "/u/cmishra1/library/anaconda3/lib/python3.5/threading.py", line 844, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread

I'm not sure how this problem is arising. I tried to understand this segment of the code:

1. Task._start_check_timer is initially called and starts a thread which will call Task.check_complete in 2.5 seconds
2. Task.check_complete checks if if Task.local_not_complete and Task.remote_not_complete, if either is not Task._start_check_timer is called again, spawns a new thread, and check_complete returns immediately after killing the thread.

That doesn't seem like it should unsustainably explode the number of threads.

Any idea what could be causing this?