All-Hands-AI
Tools for limiting tokens/requests to LLMs
What problem or use case are you trying to solve? It's easy to put OpenDevin into a loop, causing it to potentially incur huge costs from the LLM provider.
We should at least expose max_iterations. Maybe max_tokens_per_request too
Describe the UX of the solution you'd like Probably a settings modal?
Do you have thoughts on the technical implementation? Will need to ingest this in session.py, then pass it to the controller
Where do we use max_tokens_per_request? It seems I didn't find it in our codebase.
There's no max_tokens_per_request yet! Would be great to have
Probably needs to be max_chars_per_request though since tokenization happens on the LLM-side
It's easy to put OpenDevin into a loop, causing it to potentially incur huge costs from the LLM provider.
Agree. This is my first impression of OpenDevin XD. My first use of OpenDevin led to an infinite loop that I had to manually kill.
I am working on max_chars_per_request as my hopefully first contribution to opendevin.