[BUG] CrewAI 0.83 -> 0.86: Issues executing tools

[arnaudbreton]

Description

Upgrading to CrewAI 0.86 leads to issue in tool execution which is working as expected in CrewAI 0.83.

Steps to Reproduce

1. Set OPENAI_API_KEY or ULTRAVOX_API_KEY env var
2. Install the project dependencies: pip install -r requirements.txt
3. Run Fine voicing in ultravox branch https://github.com/Fine-Voicing/fine-voicing-crewai/tree/ultravox : python src/main.py
4. Observe the conversation being generated using either OpenAI or Ultravox API as voice models
5. Upgrade from CrewAI 0.83 to 0.86
6. Repeat steps 1-4
7. Observe the error mentioned below

Expected behavior

Usage of tools like in CrewAI 0.83

Screenshots/Code snippets

Most of the logic lives here: https://github.com/Fine-Voicing/fine-voicing-crewai/blob/ultravox/src/fine_voicing/test_runner.py#L111-L230

Output in CrewAI 0.83:

# Agent: Voice AI Model Agent
## Thought: I need to generate a response from Miguel to address the real estate agency representative's questions and concerns about Sandra, ensuring to highlight the benefits and security features of the assistant while also encouraging them to share the contact details.
## Using tool: UltraVox Voice AI API Client
## Tool Input: 
"{\"role_name\": \"Commercial Representative\", \"last_message\": \"Claro! A Sandra \\u00e9 projetada para facilitar a gest\\u00e3o de chamadas e compromissos com v\\u00e1rias funcionalidades. Ela pode automatizar agendamentos, realizar lembretes e at\\u00e9 responder a perguntas frequentes dos clientes, garantindo que voc\\u00ea e a sua equipe possam se concentrar em tarefas mais importantes. Em termos de seguran\\u00e7a, temos um forte sistema de prote\\u00e7\\u00e3o de dados e todas as informa\\u00e7\\u00f5es sens\\u00edveis s\\u00e3o criptografadas. Quanto \\u00e0 forma\\u00e7\\u00e3o, \\u00e9 bem simples! Oferecemos um treinamento breve que permite \\u00e0 sua equipe se familiarizar rapidamente com a Sandra e suas capacidades. Fico \\u00e0 disposi\\u00e7\\u00e3o para sua equipe experimentar a assistente e ver como ela pode se integrar aos seus processos atuais! Posso saber o seu nome e o n\\u00famero de contacto do tomador de decis\\u00f5es da sua ag\\u00eancia? Isso ajudar\\u00e1 a garantir que todos os detalhes sejam abordados.\"}"
## Tool Output: 
Commercial Representative: Olá Miguel! Meu nome é Joana e sou a gerente de marketing aqui na Agência Imobiliária Lisboa. Estou sempre procurando maneiras de melhorar a eficiência da nossa equipe, então estou curiosa para saber mais sobre a Sandra.

Pode me explicar como a Sandra funciona e como ela pode nos ajudar a gerenciar chamadas e compromissos? Nós recebemos muitas ligações diariamente e às vezes é difícil manter tudo organizado.

Além disso, o nosso diretor é o Sr. Carlos Silva, e ele é quem toma as decisões importantes aqui na agência. Seu número de contato é 213 456 789. Mas eu posso ser a pessoa de contato para discutir mais sobre a Sandra, se você preferir.

Estou ansiosa para saber mais sobre como a Sandra pode nos ajudar!

Operating System

macOS Sonoma

Python Version

3.12

crewAI Version

0.83

crewAI Tools Version

0.14

Virtual Environment

Venv

Evidence

Output in 0.83:

# Agent: Voice AI Model Agent
## Thought: I need to generate a response from Miguel to address the real estate agency representative's questions and concerns about Sandra, ensuring to highlight the benefits and security features of the assistant while also encouraging them to share the contact details.
## Using tool: UltraVox Voice AI API Client
## Tool Input: 
"{\"role_name\": \"Commercial Representative\", \"last_message\": \"Claro! A Sandra \\u00e9 projetada para facilitar a gest\\u00e3o de chamadas e compromissos com v\\u00e1rias funcionalidades. Ela pode automatizar agendamentos, realizar lembretes e at\\u00e9 responder a perguntas frequentes dos clientes, garantindo que voc\\u00ea e a sua equipe possam se concentrar em tarefas mais importantes. Em termos de seguran\\u00e7a, temos um forte sistema de prote\\u00e7\\u00e3o de dados e todas as informa\\u00e7\\u00f5es sens\\u00edveis s\\u00e3o criptografadas. Quanto \\u00e0 forma\\u00e7\\u00e3o, \\u00e9 bem simples! Oferecemos um treinamento breve que permite \\u00e0 sua equipe se familiarizar rapidamente com a Sandra e suas capacidades. Fico \\u00e0 disposi\\u00e7\\u00e3o para sua equipe experimentar a assistente e ver como ela pode se integrar aos seus processos atuais! Posso saber o seu nome e o n\\u00famero de contacto do tomador de decis\\u00f5es da sua ag\\u00eancia? Isso ajudar\\u00e1 a garantir que todos os detalhes sejam abordados.\"}"
## Tool Output: 
Commercial Representative: Olá Miguel! Meu nome é Joana e sou a gerente de marketing aqui na Agência Imobiliária Lisboa. Estou sempre procurando maneiras de melhorar a eficiência da nossa equipe, então estou curiosa para saber mais sobre a Sandra.

Pode me explicar como a Sandra funciona e como ela pode nos ajudar a gerenciar chamadas e compromissos? Nós recebemos muitas ligações diariamente e às vezes é difícil manter tudo organizado.

Além disso, o nosso diretor é o Sr. Carlos Silva, e ele é quem toma as decisões importantes aqui na agência. Seu número de contato é 213 456 789. Mas eu posso ser a pessoa de contato para discutir mais sobre a Sandra, se você preferir.

Estou ansiosa para saber mais sobre como a Sandra pode nos ajudar!

Output in 0.86:

Agent: Voice AI Model Agent

Task: Use the ultravox Client tool to generate the next message in the conversation.Provide the role_name parameter of the ultravox Client tool: Commercial RepresentativeFor Ultravox, when the chat history is [EMPTY HISTORY], last_message should be a message generated from the role prompt Olá, meu nome é Pedro e sou representante comercial. Estou ligando para apresentar a Sandra, uma assistente de voz automatizada que pode ajudar sua agência imobiliária a gerenciar chamadas e agendamentos. Como você está hoje? Posso responder alguma dúvida que você tenha?.Otherwise, provide the last_message parameter of the ultravox client tool from the chat history.Chat history, each message is prefixed with a dash (-):[EMPTY HISTORY]

I encountered an error while trying to use the tool. This was the error: 'NoneType' object has no attribute 'send'. Tool UltraVox Voice AI API Client accepts these inputs: Tool Name: UltraVox Voice AI API Client Tool Arguments: {'role_name': {'description': 'The role to play in the conversation', 'type': 'str'}, 'last_message': {'description': 'The last message in the conversation', 'type': 'str'}} Tool Description: Use the UltraVox Voice AI API to generate a message

Agent: Conversation Voice Simulation Agent

Task: Generate the next message in the conversation, based on the chat history.Play role in the conversation: Real-Estate Agency Representative.Follow these instructions: Olá, estou ouvindo. O que você pode me contar sobre essa assistente de voz? Como ela pode ajudar nossa agência? Quais são os custos envolvidos e como posso integrá-la em nossos processos?.Prefix all messages with the role name: Real-Estate Agency Representative.Ensure the response is in pt-PT and adheres to the context of the conversation.Chat history, each message is prefixed with a dash (-):-

I encountered an error while trying to use the tool. This was the error: 'NoneType' object has no attribute 'send'. Tool UltraVox Voice AI API Client accepts these inputs: Tool Name: UltraVox Voice AI API Client Tool Arguments: {'role_name': {'description': 'The role to play in the conversation', 'type': 'str'}, 'last_message': {'description': 'The last message in the conversation', 'type': 'str'}} Tool Description: Use the UltraVox Voice AI API to generate a message. Moving on then. I MUST either use a tool (use one at time) OR give my best final answer not both at the same time. To Use the following format:

Thought: you should always think about what to do Action: the action to take, should be one of [OpenAI Voice AI API Client, UltraVox Voice AI API Client] Action Input: the input to the action, dictionary enclosed in curly braces Observation: the result of the action ... (this Thought/Action/Action Input/Result can repeat N times) Thought: I now can give a great answer Final Answer: Your final answer must be the great and the most complete as possible, it must be outcome described

Action 'OpenAI Voice AI API Client' don't exist, these are the only available Actions:

Agent: Conversation Voice Simulation Agent

Using tool: OpenAI Voice AI API Client

Tool Input:

"{"role_name": "Real-Estate Agency Representative", "last_message": "Ol\u00e1, estou ouvindo. O que voc\u00ea pode me contar sobre essa assistente de voz? Como ela pode ajudar nossa ag\u00eancia? Quais s\u00e3o os custos envolvidos e como posso integr\u00e1-la em nossos processos?"}"

Tool Output:

I encountered an error: Action 'OpenAI Voice AI API Client' don't exist, these are the only available Actions:

Moving on then. I MUST either use a tool (use one at time) OR give my best final answer not both at the same time. To Use the following format:

Thought: you should always think about what to do Action: the action to take, should be one of [] Action Input: the input to the action, dictionary enclosed in curly braces Observation: the result of the action ... (this Thought/Action/Action Input/Result can repeat N times) Thought: I now can give a great answer Final Answer: Your final answer must be the great and the most complete as possible, it must be outcome described

Action 'UltraVox Voice AI API Client' don't exist, these are the only available Actions:

Possible Solution

Remain in 0.83 for now

Additional context

Project: Fine Voicing https://github.com/Fine-Voicing/fine-voicing-crewai

Goal: Generate test conversations with Voice AI models like OpenAI Realtime or Ultravox

Context:

• 3 Agents: Voice AI Model, Conversation Simulation, Moderator
• 3 Tasks: Generate each side of the conversation, Moderate
• 2 Tools: OpenAI Realtime API, Ultravox

[gvieira]

Hey @arnaudbreton! I tried running the project here and found no error. One thing I did, though, was to also upgrade crewai-tools to 0.17.0. Would you mind trying that as well?

[arnaudbreton]

Hey @gvieira, thanks for the quick response and testing it on your end!

I'm also running crewai-tools in 0.17.0 according to pip freeze. I did install get it as part of the update pip install crewai ---upgrade.

Are you running the version in the ultravox branch?

[gvieira]

@arnaudbreton I actually tried the openai branch, as I don't have a ultravox API key.

[bhancockio]

Hey @arnaudbreton !

It looks like your agents are trying to call tools that don't exist.

Action: the action to take, should be one of [OpenAI Voice AI API Client, UltraVox Voice AI API Client]

When I look at your code, here are the tool names I see:

• UltraVox API Client
• OpenAI Realtime API Client

Also, out of curiosity, can you please try setting result_as_answer=False on your tools and let me know if that changes anything?

[arnaudbreton]

Hey @bhancockio thanks for giving it a try! Indeed updating to match the name more exactly seems to do the trick with result_as_answer False or True. Would you advise I set it to True instead?

Also, any idea why it was working fine in 0.83?

Thanks for the quick resolution

[bhancockio]

Great!

And result_as_answer=True exits the core loop that agents go through when generating a response as soon as the tool returns a result.

If that's what you're aiming for, you can keep it as true. If you'd like the agent to act on the result of the tool, I'd set it to false.

And, I believe the root issue is that you have memory=true and it's remembering the name of previous tool names. Or, did you do training at any point on this crew? Training can also keep some historical records of what tools to use so maybe that could be it as well.

To start from a clean slate, I'd clear all of your memories for this crew with crewai reset-memories --all

I'm not sure why the update triggered this issue.

[arnaudbreton]

Thanks !

And result_as_answer=True exits the core loop that agents go through when generating a response as soon as the tool returns a result.

Indeed, it's the behavior I'm looking for.

And, I believe the root issue is that you have memory=true and it's remembering the name of previous tool names. Or, did you do training at any point on this crew? Training can also keep some historical records of what tools to use so maybe that could be it as well.

Thanks for pointing this out. The code got generated by your GPT when I got started and I didn't notice it! I didn't know about the reset-memories command as well. I just ran it and it worked.

Now, I'm actually running into the same issue again, where it works on 0.83 but gives the error on 0.86:

I encountered an error: Action 'OpenAI Voice AI API Client' don't exist, these are the only available Actions:

However, this tool is declared here: https://github.com/Fine-Voicing/fine-voicing-crewai/blob/ultravox/src/fine_voicing/tools/voice_ai.py#L36

And provided to the task here: https://github.com/Fine-Voicing/fine-voicing-crewai/blob/ultravox/src/fine_voicing/test_runner.py#L171

[arnaudbreton]

Is it expected that the reset-memories command requires an OPENAI API key?

crewai reset-memories --all An unexpected error occurred: Please provide an OpenAI API key. You can get one at https://platform.openai.com/account/api-keys

[arnaudbreton]

Here is what I did:

• Set memory=False across all 4 agents
• Ran crewai reset-memories --all with success

Attached, the full agent execution log using this commit fine-voicing-0.86.log

Here are additional informations I gathered:

The result (TaskOutput > raw) of the task is actually including the Voice AI Model Agent agent chain of thought:

2024-12-06 19:30:21 - sandra-outgoing - DEBUG - Commercial Representative: Muito obrigado, [Seu Nome], pela sua abertura e interesse. Vou entrar em contacto com o [Nome do Responsável] para agendar uma reunião e discutir todos os detalhes, incluindo a integração, suporte técnico e custos. Também enviarei as referências das outras agências que mencionamos. Estou confiante de que a Sandra será uma grande aliada para a sua agência imobiliária. 

Agradeço a sua disponibilidade e estou ansioso para falar consigo novamente. Tenha um excelente dia!


You ONLY have access to the following tools, and should NEVER make up tools that are not listed here:

Tool Name: OpenAI Voice AI API Client
Tool Arguments: {'role_name': {'description': 'The role to play in the conversation', 'type': 'str'}, 'last_message': {'description': 'The last message in the conversation', 'type': 'str'}}
Tool Description: Use the OpenAI Voice AI API to generate a message
Tool Name: Ultravox Voice AI API Client
Tool Arguments: {'role_name': {'description': 'The role to play in the conversation', 'type': 'str'}, 'last_message': {'description': 'The last message in the conversation', 'type': 'str'}}
Tool Description: Use the Ultravox Voice AI API to generate a message

Use the following format:

Thought: you should always think about what to do
Action: the action to take, only one name of [OpenAI Voice AI API Client, Ultravox Voice AI API Client], just the name, exactly as it's written.
Action Input: the input to the action, just a simple python dictionary, enclosed in curly braces, using " to wrap keys and values.
Observation: the result of the action

Once all necessary information is gathered:

Thought: I now know the final answer
Final Answer: the final answer to the original input question

Which in turn seems to confuse the Conversation Voice Simulation Agent:

# Agent: Conversation Voice Simulation Agent
## Task: Generate the next message in the conversation, based on the chat history.Play role in the conversation: Real-Estate Agency Representative.Follow these instructions: Olá, sou o representante da agência imobiliária. Estou ouvindo sobre a Sandra e como ela pode nos ajudar. Quais são alguns benefícios que você pode apresentar? Tenho algumas questões sobre como essa tecnologia funciona e se realmente pode atender às nossas necessidades. Também, antes de prosseguir, quem eu deveria contatar para obter mais informações?.Prefix all messages with the role name: Real-Estate Agency Representative.Ensure the response is in pt-PT and adheres to the context of the conversation.Chat history, each message is prefixed with a dash (-):- 
I encountered an error while trying to use the tool. This was the error: 'NoneType' object has no attribute 'send'.
 Tool Ultravox Voice AI API Client accepts these inputs: Tool Name: Ultravox Voice AI API Client
Tool Arguments: {'role_name': {'description': 'The role to play in the conversation', 'type': 'str'}, 'last_message': {'description': 'The last message in the conversation', 'type': 'str'}}
Tool Description: Use the Ultravox Voice AI API to generate a message.
Moving on then. I MUST either use a tool (use one at time) OR give my best final answer not both at the same time. To Use the following format:

Thought: you should always think about what to do
Action: the action to take, should be one of [OpenAI Voice AI API Client, Ultravox Voice AI API Client]
Action Input: the input to the action, dictionary enclosed in curly braces
Observation: the result of the action
... (this Thought/Action/Action Input/Result can repeat N times)
Thought: I now can give a great answer
Final Answer: Your final answer must be the great and the most complete as possible, it must be outcome described


 

Action 'OpenAI Voice AI API Client' don't exist, these are the only available Actions:

Is there a way to capture the task didn't succeed?

[arnaudbreton]

@bhancockio @gvieira thanks!

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.