Add list of agents as a possible result of speaker selection method

[heyitsaamir]

Why are these changes needed?

This change adds a new returnable type as part of the speaker_selection_method Callable. Currently, that callable allows the following returnables:

1. an Agent class, it must be one of the agents in the group chat.
2. a string from ['auto', 'manual', 'random', 'round_robin'] to select a default method to use.
3. None, which would terminate the conversation gracefully.

In this PR, we add a list[Agent] as a returnable type. This allows the speaker selection to use a list of agent as candidate agents for the group manager to select from. The associated issue describes this issue, but I'll paste the scenario here:

In my use-case, we have an Agent (agent_a) that can speak to (agent_b, agent_c, and agent_d). It should only talk to agent_d under certain specific circumstances. Otherwise, it should pick between agent_b and agent_c, without me having to make that decision (i.e. I would like the group manager to pick between the two).

The alternative to this PR would basically be to do the selection yourself in the speaker_selection_method. Something like this:

def custom_speaker_selection_func(
            last_speaker: Agent, groupchat: GroupChat
        ) -> Union[Agent, str, None]:
            selector = ConversableAgent("SelectorAgent", llm_config=self.llm_config)
            select_speaker_messages = groupchat.messages.copy()
           # The following is copied from _prepare_and_select_agents
            # If last message is a tool call or function call, blank the call so the api doesn't throw
            if select_speaker_messages[-1].get("function_call", False):
                select_speaker_messages[-1] = dict(select_speaker_messages[-1], function_call=None)
            if select_speaker_messages[-1].get("tool_calls", False):
                select_speaker_messages[-1] = dict(select_speaker_messages[-1], tool_calls=None)
            agent = groupchat._auto_select_speaker(last_speaker, selector, select_speaker_messages, [agent_b, agent_c])
            if agent:
                return agent
            return 'auto'

A much more elegant solution would be to use the proposal in this PR.

Related issue number

Closes #3215

Checks

• [x] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally. (I
• [x] I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [x] I've made sure all auto checks have passed.

[LittleLittleCloud]

return a list of agents from custom_speaker_selection_func feel not right to me..

Since you already return a list of candiates from custom_speaker_selection_func, is it possible to make a further step and choose the right next speaker from these candidates?

[heyitsaamir]

return a list of agents from custom_speaker_selection_func feel not right to me..

Since you already return a list of candiates from custom_speaker_selection_func, is it possible to make a further step and choose the right next speaker from these candidates?

speaker_selection_method as is doesn't allow sending a list of candidates. If I want to choose the next speaker, I need to explicitly choose an agent. But in my use case, I want to basically have the group manager choose between a more controlled set of candidates.

Is there a different way to go about accomplishing what my use case currently states?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 50.00000% with 5 lines in your changes missing coverage. Please review.

Project coverage is 37.81%. Comparing base (6279247) to head (2f73276). Report is 49 commits behind head on main.

Files	Patch %	Lines
autogen/agentchat/groupchat.py	50.00%	2 Missing and 3 partials :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3224      +/-   ##
==========================================
+ Coverage   32.90%   37.81%   +4.90%     
==========================================
  Files          94       97       +3     
  Lines       10235    10822     +587     
  Branches     2193     2485     +292     
==========================================
+ Hits         3368     4092     +724     
+ Misses       6580     6362     -218     
- Partials      287      368      +81     

Flag	Coverage Δ
unittests	37.77% <50.00%> (+4.86%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard. Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
10404695	Triggered	Generic High Entropy Secret	71054e3f411c874ccd43e4259681397e957c7c45	test/oai/test_utils.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[sonichi]

@yiranwu0 this is an extension of StateFlow. What do you think?

[rajan-chari]

@sonichi , @heyitsaamir Is the basic issue that the list of agents to consider for group chat depends on the state of current conversation? In group chat manager there isn't a way to filter the list of candidate agents that will go next.

[heyitsaamir]

@sonichi , @heyitsaamir Is the basic issue that the list of agents to consider for group chat depends on the state of current conversation? In group chat manager there isn't a way to filter the list of candidate agents that will go next.

So the group chat manager can receive a allowed_or_disallowed_speaker_transitions parameter where I can say what transitions are allowed. (i.e. agent_a can transition to agent_b, agent_c and agent_d). We also have a speaker_selection_method which gives me finer control to say after agent_a speaks, either let the group manager decide based on the selection strategies (auto, roundrobin etc), or specifically choose a particular agent, or end the conversation. However, in my use case, I want to basically say after agent_a speaks, I definitely do not want agent_d to talk, but I want the group manager to decide between agent_b and agent_c. I can do that with speaker_selection_method by introducing an LLM call inside it so that I tell the LLM to decide between a smaller subset of agents. OR I can modify speaker_selection_method to return a list of agents and the framework knows that when a list of agents is returned, the group manager can decide between those returned agents.

[heyitsaamir]

ping @yiranwu0 @sonichi

[joshkyh]

So the group chat manager can receive a allowed_or_disallowed_speaker_transitions parameter where I can say what transitions are allowed. (i.e. agent_a can transition to agent_b, agent_c and agent_d). We also have a speaker_selection_method which gives me finer control to say after agent_a speaks, either let the group manager decide based on the selection strategies (auto, roundrobin etc), or specifically choose a particular agent, or end the conversation. However, in my use case, I want to basically say after agent_a speaks, I definitely do not want agent_d to talk, but I want the group manager to decide between agent_b and agent_c.

Hey @heyitsaamir, wouldn't that be achieved by setting the following? allowed_or_disallowed_speaker_transitions = agent_a can transition to agent_b, agent_c (Notice A -> A and A -> D is missing) speaker_selection_method = 'auto' speaker_transitions_type = 'allowed'

[heyitsaamir]

So the group chat manager can receive a allowed_or_disallowed_speaker_transitions parameter where I can say what transitions are allowed. (i.e. agent_a can transition to agent_b, agent_c and agent_d). We also have a speaker_selection_method which gives me finer control to say after agent_a speaks, either let the group manager decide based on the selection strategies (auto, roundrobin etc), or specifically choose a particular agent, or end the conversation. However, in my use case, I want to basically say after agent_a speaks, I definitely do not want agent_d to talk, but I want the group manager to decide between agent_b and agent_c.

Hey @heyitsaamir, wouldn't that be achieved by setting the following? allowed_or_disallowed_speaker_transitions = agent_a can transition to agent_b, agent_c (Notice A -> A and A -> D is missing) speaker_selection_method = 'auto' speaker_transitions_type = 'allowed'

But I do want to allow agent_a to agent_d under certain specific circumstances. Are you saying that the speaker_selection_method can send agent_d under those conditions? And by default agent_a -> agent_b, agent_c?

[heyitsaamir]

So the group chat manager can receive a allowed_or_disallowed_speaker_transitions parameter where I can say what transitions are allowed. (i.e. agent_a can transition to agent_b, agent_c and agent_d). We also have a speaker_selection_method which gives me finer control to say after agent_a speaks, either let the group manager decide based on the selection strategies (auto, roundrobin etc), or specifically choose a particular agent, or end the conversation. However, in my use case, I want to basically say after agent_a speaks, I definitely do not want agent_d to talk, but I want the group manager to decide between agent_b and agent_c.

Hey @heyitsaamir, wouldn't that be achieved by setting the following? allowed_or_disallowed_speaker_transitions = agent_a can transition to agent_b, agent_c (Notice A -> A and A -> D is missing) speaker_selection_method = 'auto' speaker_transitions_type = 'allowed'

But I do want to allow agent_a to agent_d under certain specific circumstances. Are you saying that the speaker_selection_method can send agent_d under those conditions? And by default agent_a -> agent_b, agent_c?

This does work and gets me past my particular issue. So this update isn't completely necessary (for my use-case).

The benefit of this that if your agent needs to choose between two agents (agent_b, agent_c) for some cases and (agent_d, agent_e) for others. I guess then a different strategy needs to be adopted? If that is the case, then we can close this out. Otherwise, I would advocate for merging this in :).