llamabot
llamabot copied to clipboard
ericmjl
Issues with recreating Agentbot example
Hi @ericmjl, I was having some issues using the Agentbot and decide to try and recreate this example (https://ericmjl.github.io/llamabot/tutorials/agentbot/). Running the identical code as steps 1, 2 and 3 I get the following error (AgentBot_error_gpt4.txt). Running the same code but with openrouter/anthropic/claude-2 gives a similar error(AgentBot_error_claude2.txt).
Looks like an issue with how the requests are passed onto the models but haven't tried much intensive debugging yet.
Many thanks, Kalen
AgentBot_error_claude2.txt AgentBot_error_gpt4.txt
Hi @KalenJosifovski, thanks for raising the question!
It looks like GPT-4 might not support the particular format that LiteLLM (which I'm using underneath the hood for farming out to various API and local LLM providers) uses for tool calling. I'd need to do some digging on that one though, as I might be wrong. The model landscape has evolved very quickly, so to give myself a stable base to experiment with I've stuck mostly with gpt-4o.
As for the use of claude via openrouter, I have to admit that I don't have much experience with openrouter. I can definitely experiment with claude directly via my own API key and see what happens.
@ericmjl np, Im pretty new to this so my own debugging attempts didn't yield many fruitful results however I did discover the following; openAI's function call and structured outputs API make use of JSON Schema Draft 7. As of last year I believe this schema doesn't support undefined types. So ToolsToCall, where the function parameters are defined as type any, may be causing the issue.
Disclaimer: I may be wrong about the above but thats how I understood it.
Also, using GTP-4o, did you manage to run your example AgentBot code successfully or did you have the same issue that I reported?
Yes, with GPT-4o, the examples work in my hands. Perhaps it may help to give that a shot?
Thanks to your issue report, and combining it with my own experiences, I realized I needed to do a bit of additional validation of model names to ensure that anly models that support structured generation, which is a prerequisite for AgentBot's behavior, are allowed to be used when using AgentBot. That PR has now been merged and I will be cutting a new patch release of LlamaBot shortly.
Thats interesting. For me got-4o gives the same error. Perhaps it's a change in the openAI api. What versions of the required python libraries are you using?
I'll have a look to see if this new patch fixes things using the get-4o model. Did you have a look at openAI's change of type hint requirements?
Also I had a thought about your note that agentbot should include a planning step and it reminded me of this (https://github.com/Doriandarko/RAT-retrieval-augmented-thinking). Using deepseek-r1 to generate a plan before feeding that to other models. Might be useful for implementing a planning step. Not sure if it's a practical solution; I'm currently doing a course transition to data science so fairly new around here. However in future I'd love to be more involved in llamabot. I think scientific agents could be a great potential use case.
Thats interesting. For me got-4o gives the same error. Perhaps it's a change in the openAI api. What versions of the required python libraries are you using?
Hmmm... that's interesting indeed. Let me think about it. Meanwhile, I've done yet a few more updates to llamabot. Could you try those and see what happens?