haystack
haystack copied to clipboard
Fix: Add rate limit to OpenAIAnswerGenerator, change token computation
Related Issues
- fixes #3068
- A related PR adjusting the temperatur param as well: https://github.com/deepset-ai/haystack/pull/3073
Proposed Changes:
- Added a rate limit since otherwise the API will crash eventually.
- Added a try catch around the API call since OpenAIs API sometimes returns errors
- Changed prompt length computation, since one has to take the output length into account as well
How did you test it?
manual verification
Checklist
- [x] I have read the contributors guidelines and the code of conduct
- [ ] I have updated the related issue with new insights and changes
- [ ] I added tests that demonstrate the correct behavior of the change
- [x] I've used the conventional commit convention for my PR title
- [x] I documented my code
- [x] I ran pre-commit hooks and fixed any issue
Sorry, I realize I won't be able to work on this the next week. I can either look into it afterward or you start taking over?
Closing this in favor of https://github.com/deepset-ai/haystack/pull/3398