stanfordnlp
[Feature] Async Context Manager Support for Settings
What feature would you like to see?
It would be great to have the ability to use asyncify within dspy.settings.context(lm=model) by implementing an asynchronous version of the context manager. Currently seeing race conditions using asyncify with this context manager.
Would you like to contribute?
- [ ] Yes, I'd like to help implement this.
- [x] No, I just want to request it.
Additional Context
No response
@jd02904 Thank for reporting the issue! This shouldn't be the case, could you share a reproducible code? Thank you!
@chenmoneygithub I checked the source code and it seems the way DSPy handles race conditions on settings is through thread_local_overrides but thread-local storage would only work for threads, not coroutines (asyncify). So it would make sense for there to be a potential for race conditions on settings if two different coroutines write to it. Let me know if I am missing something.
Let me try to build a minimal reproducible example but might be tricky because this only happens in production with a high number of parallel requests.
For some context - asyncify builds on top of the fake streaming, which essentially uses multithreading. Our code could indeed cause an issue on native async, and I am working on the fixes. My question was to clarify which path you are using.
I have same issue on this. All our code using async with dspy.context and the code have different data to context object. The current threading model won't work for couroutine
