Connecting agency-swarm to own gpt-like LLM API like the new Llama-3-70B-Instruct

[NVS-AI]

So, as the title says. I've been wondering how difficult would it be to replace the openai API with connector to a Llama instance running somewhere, potentially even locally (I do have the compute power). Seems at first glance the API integration is quite hot-swappable, but I wanted to get the creator's thoughts on the matter. Having an option like this could mean significant performance upgrade and cost savings.

[samuelmukoti]

I’ve heard the creator speak many times on his YouTube about why he only will support OpenAI right now. He believes in building upon the best framework and tools available and OpenAI is the leader right now, he also mentioned how he leverages the OpenAI assistance API which is specific and I’m not sure if supported by other LLMs weather local or cloud.

Sam

[codermrrob]

I heard in another recent YouTube video today that @VRSEN is looking into this. It would be very cool, OpenAI is restrictively expensive for solo developer to experiment with :( and while I have been making great progress I could easily run up bills of 100+ a week building out and refining an agency. Having a cheaper option for learning and experimenting with I think is going to be important for this framework to gain traction, as good as it is.

[VRSEN]

Hey everyone, I'm currently testing the repo https://github.com/datastax/astra-assistants-api, which worked with Assistants v1. However, it may need some time to update to v2. Nonetheless, for any project that mimics the Assistants API, you just need to replace the OpenAI client like this:

import openai
from agency_swarm import set_openai_client
client = openai.OpenAI(api_key="whatever", base_url="http://127.0.0.1:8000/")
set_openai_client(client)

That should literally be it, if it's fully compatible.

If you manage to get any of these repos working, please ping me, and I'll give you a shoutout on the next video.

[NVS-AI]

If you manage to get any of these repos working, please ping me, and I'll give you a shoutout on the next video.

Thanks, will do :) Currently playing around with starting a Llama 3 instance locally. then gonna see about a wrapper API... Worst case scenario, will make one that mimics the openai API.

[jay377]

Author

Yes please somebody make this work, i actually want to learn a real agents tool that's production capable but to poor to experiment on openai

[slavakurilyak]

@VRSEN astra-assistants-api currently has 82% coverage for Assistants v1 (Assistants v2 support is planned here)

In the meanwhile, check out MLT-OSS/open-assistant-api and https://github.com/BerriAI/litellm/issues/2842 (planned feature)

[Ga68]

Thanks, will do :) Currently playing around with starting a Llama 3 instance locally. then gonna see about a wrapper API... Worst case scenario, will make one that mimics the openai API.

@NVS-AI ¿any luck? I was excited to see something taking a stab at this.

[VRSEN]

Got this to work flawlessly with any llms https://github.com/VRSEN/open-assistant-api You can even use different llms for different agents

Video coming up shortly...

[briancg42]

Thank you @VRSEN, this is a huge step ⚡

Got this to work flawlessly with any llms https://github.com/VRSEN/open-assistant-api You can even use different llms for different agents

Video coming up shortly...

[codermrrob]

Great work @VRSEN - It's exactly what this space needs. Can't wait to try it out. Especially extendable built in tools, different llms for different agents, and wondering possibility for a separate llm/model for tool selection? There seems to be a lot of work happening for this and smaller specific models look like they may perform better than larger more general llms when it comes to accurate tool selection. Guess I need to go read the code now :)

[phact]

@codermrrob separate LLM for tool selection is something we've been thinking about a bunch. I went ahead and created https://github.com/datastax/astra-assistants-api/issues/28

We'll prioritize it after we have support for assistants v2.

[slavakurilyak]

@VRSEN Excellent work on open-assistant-api

By the way, litellm also added support for openai assistants

[VRSEN]

Hey @slavakurilyak, yes, I saw that, however we need it to be supported in their proxy server for seamless integration. Once that's done I will update the docs here: https://vrsen.github.io/agency-swarm/advanced-usage/open-source-models/

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Please upgrade to the latest version and test it again.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale. If the issue still persists, please reopen.