stanfordnlp
[Bug] Runtime error scheduling new features
What happened?
What I was trying to do I’m running a DSPy GEPA optimization to tune a “scope extraction” program. The metric calls another small DSPy module (CustomSemanticF1 using ChainOfThought) that compares prediction vs gold. My LM is Azure OpenAI, configured via DSPy (which uses LiteLLM under the hood).
What actually happened (I think) DSPy spins up background worker threads to evaluate examples faster. When GEPA/evaluation is wrapping up, those workers get shut down. But LiteLLM (from my custom metric calling ChainOfThought) is still trying to submit one more small task to the same thread pool. Because the pool is already closed thus causing the error. So then python raises RuntimeError: cannot schedule new futures after shutdown
Here is the full error: ... File ".../dspy/clients/lm.py", line 421, in litellm_responses_completion return litellm.responses( ^^^^^^^^^^^^^^^^^^ File ".../litellm/utils.py", line 1356, in wrapper raise e File ".../litellm/utils.py", line 1287, in wrapper executor.submit( File "/usr/lib/python3.12/concurrent/futures/thread.py", line 170, in submit raise RuntimeError('cannot schedule new futures after shutdown') RuntimeError: cannot schedule new futures after shutdown
Steps to reproduce
Code to run:
import dspy
lm = dspy.LM( "azure/gpt-5-mini", # example deployment api_key="", api_base="", api_version="2025-03-01-preview", model_type="responses", temperature=1.0, max_tokens=2000, ) dspy.configure(lm=lm)
class ExtractScopeFromChunks(dspy.Module): def forward(self, docs_chunks): # trivial predictor to reproduce with dspy.context(lm=lm): out = dspy.Predict(lambda docs_chunks: {"response": "mock"})(docs_chunks=docs_chunks) return dspy.Prediction(response=out.response)
student = ExtractScopeFromChunks()
Metric: calls a ChainOfThought internally (triggering extra LM calls)
class CustomSemanticF1(dspy.Module): def init(self): self.module = dspy.ChainOfThought(lambda question, ground_truth, system_response: {"precision": 1.0, "recall": 1.0, "discussion": ""}) def forward(self, example, pred, trace=None): r = self.module(question=example.question, ground_truth=example.response, system_response=pred.response) f1 = (r["precision"] * r["recall"] * 2) / (r["precision"] + r["recall"]) return f1, r["discussion"], [], []
sf1 = CustomSemanticF1()
def gepa_metric(gold, pred, **_): f1, discussion, _, _ = sf1(gold, pred) return dspy.Prediction(score=f1, feedback=discussion)
devset = [ dspy.Example(docs_chunks=[["x"]], question="q", response="gold").with_inputs("docs_chunks"), dspy.Example(docs_chunks=[["y"]], question="q", response="gold").with_inputs("docs_chunks"), ]
gepa = dspy.GEPA(metric=gepa_metric, track_stats=True, reflection_lm=lm, max_metric_calls=64) best_program = gepa.compile(student=student, trainset=devset, valset=[])
Steps to Reproduce: (1) Run the code above (2) Observe near the end of the code run/optimization the runtime error.
DSPy version
dspy-ai: 3.0.3
@mwilliamsempower Thanks for reporting the issue!
File ".../litellm/utils.py", line 1356, in wrapper
raise e
File ".../litellm/utils.py", line 1287, in wrapper
executor.submit(
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 170, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Seems this future scheduling happens on the litellm side, which is a bit strange. Could you create an issue report in litellm and link to this issue? I have never seen this before with litellm.completion, so I am suspecting it's some strange behavior with responses API. Btw would you mind adding description to the title? thank you!
Here is issue report with litellm : https://github.com/BerriAI/litellm/issues/15366
Thanks for your help!
I wanted to provide some more context this is the full traceback and code I see before that happens. I think there is a possibility the error is coming from the dspy side.
2025/09/29 12:29:04 WARNING dspy.adapters.json_adapter: Failed to use structured output format, falling back to JSON mode. 2025/09/29 12:30:15 ERROR dspy.utils.parallelizer: Error for Example({'docs_chunks': [['PERFORMANCE WORK STATEMENT (PWS) \nDEPARTMENT OF VETERAN S AFFAIRS \nOffice of Information & Technology \nCompliance, Risk and Remediation \n \nFreedom of Information Act Compliance \n \nDate: September 3, 2025 \n \nVA-25-00094887 \n \n \nPWS Version Number: 3.3\n\nFreedom of Information Act Compl
Traceback (most recent call last): File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\adapters\chat_adapter.py", line 38, in call return super().call(lm, lm_kwargs, signature, demos, inputs) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\adapters\base.py", line 127, in call outputs = lm(messages=inputs, **lm_kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\utils\callback.py", line 326, in sync_wrapper return fn(instance, *args, **kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\base_lm.py", line 85, in call response = self.forward(prompt=prompt, messages=messages, **kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\lm.py", line 147, in forward results = completion( request=dict(model=self.model, messages=messages, **kwargs), num_retries=self.num_retries, cache=litellm_cache_args, ) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\cache.py", line 235, in sync_wrapper result = fn(*args, **kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\lm.py", line 421, in litellm_responses_completion return litellm.responses( ~~~~~~~~~~~~~~~~~^ cache=cache, ^^^^^^^^^^^^ ...<2 lines>... **request, ^^^^^^^^^^ ) ^ File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\litellm\utils.py", line 1344, in wrapper raise e File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\litellm\utils.py", line 1275, in wrapper executor.submit( ~~~~~~~~~~~~~~~^ logging_obj.success_handler, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... end_time, ^^^^^^^^^ ) ^ File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\concurrent\futures\thread.py", line 171, in submit raise RuntimeError('cannot schedule new futures after shutdown') RuntimeError: cannot schedule new futures after shutdown
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\adapters\json_adapter.py", line 78, in call return super().call(lm, lm_kwargs, signature, demos, inputs) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\adapters\chat_adapter.py", line 46, in call raise e File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\adapters\chat_adapter.py", line 38, in call return super().call(lm, lm_kwargs, signature, demos, inputs) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\adapters\base.py", line 127, in call outputs = lm(messages=inputs, **lm_kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\utils\callback.py", line 326, in sync_wrapper return fn(instance, *args, **kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\base_lm.py", line 85, in call response = self.forward(prompt=prompt, messages=messages, **kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\lm.py", line 147, in forward results = completion( request=dict(model=self.model, messages=messages, **kwargs), num_retries=self.num_retries, cache=litellm_cache_args, ) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\cache.py", line 235, in sync_wrapper result = fn(*args, **kwargs) File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\dspy\clients\lm.py", line 421, in litellm_responses_completion return litellm.responses( ~~~~~~~~~~~~~~~~~^ cache=cache, ^^^^^^^^^^^^ ...<2 lines>... **request, ^^^^^^^^^^ ) ^ File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\litellm\utils.py", line 1344, in wrapper raise e File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\site-packages\litellm\utils.py", line 1275, in wrapper executor.submit( ~~~~~~~~~~~~~~~^ logging_obj.success_handler, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... end_time, ^^^^^^^^^ ) ^ File "C:\Users\mwilliams\AppData\Local\Programs\Python\Python313\Lib\concurrent\futures\thread.py", line 171, in submit raise RuntimeError('cannot schedule new futures after shutdown') RuntimeError: cannot schedule new futures after shutdown
