OpenHands icon indicating copy to clipboard operation
OpenHands copied to clipboard

Revert the use of Router, good ole completion works.

Open enyst opened this issue 2 years ago • 0 comments

Upon more investigation on this one, it turns out that:

  • our most active upstream, liteLLM, released a patch and fixed some issues with async calls and their Router class.(example) The worst behavior that I was seeing is gone. For reference: when you didn't have correct credentials, the errors went on in a loop in the Router constructor (meaning our LLM constructor), and were printing sensitive information. I am not sure I see clearly what exactly fixed it, but I'm not seeing that anymore.

  • updated poetry.lock to propagate this version.

  • the stop loop in agent controller doesn't work (anymore?). Hot-fixed with a string that works for openai. I am not sure if this might work also for openai-compatible setups.

  • I think we're better off reverting the use of Router, at least for now. It was added to handle retries and rate limiting, but for retries at least, it doesn't work as expected. I think its use needs more work in connection with our exception handling. I have a branch in progress on that, it needs more work though. Worth noting also that according to LiteLLM docs I am not sure we're using it as intended anyway. So this PR continues to propose that we put the use of Router for our features in the back-burner until we have tests and see it works. Although, after fighting it, I might not be a model of objectivity on that one. 🤣

enyst avatar Apr 08 '24 15:04 enyst