AgentPilot

AgentPilot copied to clipboard

hey @jbexta, thanks very much for the new release.

AP: 1.5.7
litellm(proxy): 1.17.5
ollama: 0.1.20

I'm still stocked at the model-setup.

I'm testing with

1. direct connection to together.ai: nok in the API-menu: select together.ai (provider are not alphabetical orderd)

• added key in privat key
• tried to find a model which is impossible in the API/Model-list due to no readability. (long model names)
• only a very old subset of the models are offered. missing i.e mixtral,

then setting the model in System trying to chat: pop-up error: no OPENAI_API_KEY is set

2. using ollama (on my ext. server)

• choose ollama provider
• only a fraction of the latest modellist is shown.
• the local modified models(files) are not shown.
• where can I set the API_BASE_URL for all my ollama-models at once? or do I have to set that for every model seperat, instead of provider-wide?
• adding my own model-file-name to the list.
  • setting URL
  • need to restart to see in 'System'-Menu
  • choose the new model

trying to chat: -> same OPENAI_API_KEY_error, even if I set a dummy one in ollama: private-key

4. what are the client-key and privat-key fields for?
   • guess: private-key is the api-key?
   • client-key: ?

when exporting a OPENAI_API_KEY in the env, before starting, the error changes:

litellm.exceptions.AuthenticationError: OpenAIException - Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-nobro. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}

the call get routed to openai.com, instead of using either the together-api-endpoint, ollama, or custom-proxy ones.

Suggestion: create a hirachical Settings structure:

1. BASE-Settings
2. PROVIDER-Settings query litellm for default model list & params, cost, etc.: see token_usage
3. MODEL-Settings, fill in the queried params as default overwrite as necessary

workflow 1: Use a LiteLLM provided services:

1. fill in all BASE-Settings with all neccessary params, then
2. PROVIDER-Settings: choose a provider
3. fill in all neccessary: i.e.

• API-KEY if necessary
• BASE_URL for all provider's models.
• then, query the provider for available modules <API_BASE_URL>/v1/models
• then show the recieved model-list in a wide-enough window, so one even can read replicate's and together.ai`s very long model-names. since we already choose the provider, the first field of the litellm-modelanme schema could be dropped for readability.

3. MODEL-Settings:

• then only offer model-spec overwrites, like, temp, top_?, c_ntx, etc.

workflow 2: in case of a proxy-use *), we need to set:

1. BASE-settings
2. PROVIDER-settings: create/choose: custom/proxy

• custom API_BASE_URL
• provider: API to use, like openai (for litellm-proxy) but without asking for an API_KEY or use dummy, if empty and (API_BASE_URL != api.openai.com)
• query the proxy for avail. model-(aliases)

3. set MODEL-setting overwrites

*) litellm as proxy/router/cache can hold all neccessary info like KEY, URL, provider, model-params

Caveat:

with all the remote queries, this would need to have an airplane-mode. b/c in case we run the API local (i.e. ollama & running tiny Models), we would loose usability, if we depend on online checks. therefore an online/offline switch, or maybe don't query by default and implement an update-button could be a solution. the later one would IMHO be preferable, since we can control how often we need to update the list, while tinkering away ;) the online/offline availability could also go into the provider settings?

what do you think?

I'm sorry to say, but with all these errors since 0.1.5 I still get it not to work.

1. direct connection to together.ai: **nok**
  in the `API`-menu: select together.ai (provider are not alphabetical orderd)

If you click the Name header it will sort the table, that will be default on next version

 * tried to find a model which is impossible in the API/Model-list due to no readability. (long model names)
 [...]
 * then show the recieved model-list in a wide-enough window, so one even can read replicate's and together.ai`s very long model-names. since we already choose the provider, the first field of the litellm-modelanme schema could be dropped for readability.

It uses litellm's model names, it would be possible to remove the prefix eg together_ai/.. from the model names, that would need an additional field for each API something like 'litellm prefix'. I'll get around to that but it won't be soon, I'll do that after 0.2.0 is released (within a few months)

For now long model names can be viewed and edited in the model name field next to the list in API settings, I know it's tedious, but even with the prefix removed some model names will still be cut off.

Also the list can be scrolled horizontally if you have a trackpad with horizontal scrolling, maybe I should enable the horizontal scrollbar for now.

  * only a very old subset of the models are offered. missing i.e mixtral,

Models can be added manually with the new icon, I add some new models as time goes on like GPT 4.5 turbo etc, but maintaining the collection of models from all providers is a lot of work, until there's an easy way to scrape them from litellm website or library

 then setting the model in `System` trying to chat: pop-up error: no OPENAI_API_KEY is set

Are you changing the model in the Context settings or the Agent settings? When a new chat is created from the Agents list it uses the default agent settings. But when a new chat is created using the + icon in the chat page, the agent settings is copied from that specific context, so if that context still had an openai model then it will copy it, this behaviour is needed when it comes to more complex contexts like multi agent chat, although maybe it should be clearer. So basically, each chat agent has it's own settings, the Agents page settings are the 'default agent settings'.

If the model in context settings (access from the chat page by clicking the header) is together_ai and it's still asking for openai api key, then let me know I'll investigate further, but it might be a litellm issue I'm not sure.

 2. using ollama (on my ext. server)
 * the local modified models(files) are not shown.

I've not tested ollama, but you would need to add the model names manually in agent pilot. I need to get around to testing ollama, I still haven't tried it.

 * where can I set the API_BASE_URL for all my ollama-models at once?
   or do I have to set that for every model seperat, instead of provider-wide?

For now yes it has to be set for each model

  need to restart to see in 'System'-Menu

Oops I'll see to that

   4. what are the `client-key` and `privat-key` fields for?

client key is rarely used but needed for some APIs (like AWS) private key is the api key yes, I will rename that to API key

    but without asking for an API_KEY or use dummy, if empty and (API_BASE_URL != api.openai.com)

Let me know if you still get this response after changing the context specific model (or contexts created from the agent defautls)

with all the remote queries, this would need to have an airplane-mode. b/c in case we run the API local (i.e. ollama & running tiny Models), we would loose usability, if we depend on online checks. therefore an online/offline switch, or maybe don't query by default and implement an update-button could be a solution. the later one would IMHO be preferable, since we can control how often we need to update the list, while tinkering away ;) the online/offline availability could also go into the provider settings?

This is something I need to look into, I think open interpreter had the same probem, but not sure what you mean by update button and update the list?

Btw there is a new button to update all context agents when you change the default agent settings, this will apply the new settings to all the previous chats

Hey @chymian @jbexta anything we can do to help here?

Context - i'm the maintainer of litellm

I think these issues are sorted now, feel free to reopen if you find anything else