microsoft
Pausing before passing conversation to next agent[Feature Request]:
Is your feature request related to a problem? Please describe.
The feature request is related to the issue of managing rate limits. As all are aware OpenAI limits the rate of tokens per second and it is easy to hit that limit without warning. A way to insert a pause (similar in function to the Unix "sleep" command) to allow a set pause to occur before the next agent in a conversation starts processing the output of the previous agent in the conversation would allow for a simple way of avoiding hitting the rate limits for a given account.
Describe the solution you'd like
For instance, in the "Auto Generated Agent Chat: Performs Research with Multi-Agent Group Chat" example it is easy to hit the rate limit as the Coder agent, Critic agent, and Scientist agent discuss how to implement the task of finding papers on LLM applications from arxiv and creating a markdown table of different domains.
The solution I'd like to see is a method to set a pause time, similar in functionality to the Unix "sleep" command, before the output of one agent is sent onto the next agent. This should be easier to implement than a more dynamic 'check to see current number of tokens sent during a window of.....' although ultimately this would be the best solution of all.
Additional context
No response
@sonichi Do you think this would have value? If so, where would be the sleep call live in the code, and how should the option be exposed?
This behaviour could be achieved with existing code by registering a hook that just calls sleep.
@rickyloynd-microsoft am I right?
Good point. For instance, the new AgentCapability class could be called CanSleep. It would define a hook and call register_hook to attach it to the existing hookable method, process_last_message, which gets called by generate_reply on every invocation.
Has there been any changes? I only see a discussion on how this could be implemented.
@LopezNuance this could be implemented on the application level, there is no need to change anything in the framework. If you wish, we could create an example for you.
Ok, I misunderstood the posts, sorry about that. Sure, an example would be great. Then go ahead and close the feature request. Thanks!
I see Davor Runje's comment especially applies: https://github.com/microsoft/autogen/pull/1240#issuecomment-1892502988
@davorrunje
@LopezNuance this could be implemented on the application level, there is no need to change anything in the framework. If you wish, we could create an example for you.
@davorrunje If you wouldn't mind, could you provide me with an example and then go ahead and close the feature request?