OpenHands icon indicating copy to clipboard operation
OpenHands copied to clipboard
verified publisher icon All-Hands-AI

Unable to find the ollama models in the dropdown list

Open kiran-chinthala opened this issue 1 year ago • 12 comments

I have used this below configuration for devin and ollama server started and running, on the other terminal using litellm to use the API's.

#########devin################

python -m pip install pipenv python -m pipenv install -v python -m pipenv shell uvicorn opendevin.server.listen:app --port 3000

cd frontend npm install npm start

.env or config.toml export LLM_EMBEDDING_MODEL="ollama/llama2" export LLM_API_KEY="" export LLM_MODEL="ollama/llama2" export WORKSPACE_DIR="./workspace" export LLM_BASE_URL="http://localhost:8000"

$ litellm --model ollama/llama2 --add_function_to_prompt --drop_params --request_timeout 1000 --debug --max_tokens 5000 INFO: Started server process [75584] INFO: Waiting for application startup. #------------------------------------------------------------#

'It would help me if you could add...'

https://github.com/BerriAI/litellm/issues/new

#------------------------------------------------------------#

Thank you for using LiteLLM! - Krrish & Ishaan

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new

INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Devin backend server started successfully

$ uvicorn opendevin.server.listen:app --port 3000 INFO: Started server process [79569] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:3000 (Press CTRL+C to quit) INFO: ('127.0.0.1', 63888) - "WebSocket /ws" [accepted] INFO: connection open INFO: 127.0.0.1:63890 - "GET /litellm-models HTTP/1.1" 200 OK INFO: 127.0.0.1:63890 - "GET /litellm-models HTTP/1.1" 200 OK

Frontend

$npm start

VITE v5.1.6 ready in 661 ms

➜ Local: http://localhost:3001/ ➜ Network: use --host to expose ➜ press h + enter to show help

🌼 daisyUI 4.9.0 ├─ ✔︎ 1 theme added https://daisyui.com/docs/themes ╰─ ★ Star daisyUI on GitHub https://github.com/saadeghi/daisyui

🌼 daisyUI 4.9.0 ├─ ✔︎ 1 theme added https://daisyui.com/docs/themes ╰─ ❤︎ Support daisyUI project: https://opencollective.com/daisyui

while selecting Model from UI, I am unable to see the ollama/llama2 in the given list. Is there any steps I need to follow to display local llm in the given list. image

And then I am getting lots of exceptions get choose the right LLM from the list. Please do the needful. Thanks

kiran-chinthala avatar Apr 03 '24 12:04 kiran-chinthala

It was a bug in LIiteLLM, already discussed and fixed here (it was for Gemini, but should fix other missing entries too).

https://github.com/BerriAI/litellm/pull/2806

I guess it should fix itself as soon the new build of litellm gets released (you will need to update python dependencies too)

Anyway, there are some people wich have already made sone experiments with ollama, so i guess it's possible to just put it in the config.toml. should look like this: LLM_API_KEY="11111111111111111111" WORKSPACE_DIR="workspace" LLM_BASE_URL="http://localhost:11434" LLM_MODEL= "ollama/llama2" LLM_EMBEDDING_MODEL="llama2"

You can also use something like KoboldCpp (or anything wich exposes an OpenAI-compatible API) and configure OpenDevin like this:

LLM_API_KEY="." LLM_MODEL="gpt-4-0125-preview" LLM_BASE_URL="http://localhost:5001/v1/" WORKSPACE_DIR="./workspace"

DGdev91 avatar Apr 03 '24 12:04 DGdev91

Thanks Davide for the reply, I have used both the LiteLLM and only Ollama, but the both cases I got similar response. However, I will try with these properties in config.toml file. But What Model I need to choose from Devin side to work with my local LLM.

curl http://localhost:11434 Ollama is running

kiran-chinthala avatar Apr 03 '24 13:04 kiran-chinthala

image

Should be fixed with the latest code

xcodebuild avatar Apr 03 '24 13:04 xcodebuild

Yes, I just updated to the litellm==v1.34.22, I have selected llama2 from the model list, but I am getting this error in the litellm side. below are the logs.

INFO: 127.0.0.1:61419 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61420 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61421 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61422 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61423 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61424 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61425 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61426 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61427 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61428 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61429 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61430 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61431 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61432 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61433 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61434 - "POST /api/embeddings HTTP/1.1" 404 Not Found INFO: 127.0.0.1:61435 - "POST /api/embeddings HTTP/1.1" 404 Not Found

image

kiran-chinthala avatar Apr 03 '24 13:04 kiran-chinthala

Thanks Davide for the reply, I have used both the LiteLLM and only Ollama, but the both cases I got similar response. However, I will try with these properties in config.toml file. But What Model I need to choose from Devin side to work with my local LLM.

curl http://localhost:11434 Ollama is running

Follow my examples. For ollama you should use ollama/{model_name}

While for KoboldCpp or other software wich expose an OpenAI-like API, select gpt-4-0125-preview and just set LLM_BASE_URL

DGdev91 avatar Apr 03 '24 14:04 DGdev91

I am using these settings in my config.toml:

LLM_API_KEY="11111111111111111111"
LLM_BASE_URL="http://localhost:11434"
LLM_MODEL= "ollama/llama2"
LLM_EMBEDDING_MODEL="llama2"
WORKSPACE_DIR="./workspace"

I also chose llama2 as Model and MonologueAgent as Agent on the front end.

The backend server is giving this error: litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=llama2

hasanzu514 avatar Apr 03 '24 21:04 hasanzu514

I have made a PR to try and fix this on the backend #689.

I changed the backend llm so that if the user has LLM_API_KEY="ollama" in their config then it will ignore the input that comes from the frontend. This is more of a quick hack since this makes the model dropdown menu essentially useless when you have the api key set to ollama, but if you are using local most likely you are not using the api. You can also just restart with a new config if you want to use the api.

JayQuimby avatar Apr 04 '24 03:04 JayQuimby

I have made a PR to try and fix this on the backend #689.

I changed the backend llm so that if the user has LLM_API_KEY="ollama" in their config then it will ignore the input that comes from the frontend. This is more of a quick hack since this makes the model dropdown menu essentially useless when you have the api key set to ollama, but if you are using local most likely you are not using the api. You can also just restart with a new config if you want to use the api.

Can I pull the latest code and try?

kiran-chinthala avatar Apr 04 '24 07:04 kiran-chinthala

I just tested with latest code pull, getting this error while connecting ollama config.toml LLM_API_KEY="ollama" WORKSPACE_DIR="./workspace" LLM_BASE_URL="http://localhost:4000" LLM_MODEL="ollama/llama2" LLM_EMBEDDING_MODEL="llama2"

after backend server starts then I am getting this error, I have selected llama2 and agent is monologueAgent. $ uvicorn opendevin.server.listen:app --port 3000 INFO: Started server process [55190] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:3000 (Press CTRL+C to quit) INFO: ('127.0.0.1', 61135) - "WebSocket /ws" [accepted] INFO: connection open INFO: 127.0.0.1:61136 - "GET /litellm-models HTTP/1.1" 200 OK INFO: 127.0.0.1:61137 - "GET /litellm-agents HTTP/1.1" 200 OK INFO: 127.0.0.1:61138 - "GET /litellm-models HTTP/1.1" 200 OK INFO: 127.0.0.1:61139 - "GET /litellm-agents HTTP/1.1" 200 OK

ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 240, in run_asgi result = await self.app(self.scope, self.asgi_receive, self.asgi_send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/middleware/errors.py", line 151, in call await self.app(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/middleware/cors.py", line 77, in call await self.app(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/routing.py", line 373, in handle await self.app(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/routing.py", line 96, in app await wrap_app_handling_exceptions(app, session)(scope, receive, send) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/starlette/routing.py", line 94, in app await func(session) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/fastapi/routing.py", line 348, in app await dependant.call(**values) File "/Users/Kiran/Documents/GitHub/OpenDevin/opendevin/server/listen.py", line 25, in websocket_endpoint await session.start_listening() File "/Users/Kiran/Documents/GitHub/OpenDevin/opendevin/server/session.py", line 89, in start_listening await self.create_controller(data) File "/Users/Kiran/Documents/GitHub/OpenDevin/opendevin/server/session.py", line 135, in create_controller llm = LLM(model=model, api_key=api_key, base_url=api_base) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Kiran/Documents/GitHub/OpenDevin/opendevin/llm/llm.py", line 36, in init self._router = Router( ^^^^^^^ File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/litellm/router.py", line 198, in init self.set_model_list(model_list) File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/litellm/router.py", line 2075, in set_model_list ) = litellm.get_llm_provider( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/litellm/utils.py", line 5733, in get_llm_provider raise e File "/Users/Kiran/.local/share/virtualenvs/OpenDevin-N0zELi1b/lib/python3.11/site-packages/litellm/utils.py", line 5720, in get_llm_provider raise litellm.exceptions.BadRequestError( # type: ignore litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=llama2 Pass model as E.g. For 'Huggingface' inference endpoints pass in completion(model='huggingface/starcoder',..) Learn more: https://docs.litellm.ai/docs/providers INFO: connection closed

kiran-chinthala avatar Apr 04 '24 12:04 kiran-chinthala

@kiran-chinthala my PR never got merged. Someone in the dev team told me that they are working on a better solution than the hack fix I put in my PR. For now I recommend this change as a temporary fix: in opendevin/llms/llm.py change first line of init to this:

self.model_name = model if model and api_key.lower() != 'ollama' else DEFAULT_MODEL_NAME

my current config:

LLM_API_KEY="ollama"
LLM_BASE_URL="http://localhost:11434"
LLM_MODEL="ollama/dolphin-mistral:7b-v2.8-q8_0"
LLM_EMBEDDING_MODEL="local"
WORKSPACE_DIR="./workspace"
MAX_ITERATIONS=100

JayQuimby avatar Apr 04 '24 12:04 JayQuimby

hi, i have this configuration:

LLM_MODEL="ollama/llama2:latest" LLM_API_KEY="ollama" WORKSPACE_DIR="./workspace" LLM_BASE_URL="http://localhost:11434" LLM_EMBEDDING_MODEL="local"

but it gives me these errors:

ERROR: "'action' key is not found in action={'thought': 'How should I structure my calculator program? Should I start with a main function and define functions for each feature, or should I start with a class and implement inheritance?'}" Traceback (most recent call last): File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/opendevin/controller/agent_controller.py", line 113, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/agenthub/monologue_agent/agent.py", line 166, in step action = prompts.parse_action_response(action_resp) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/agenthub/monologue_agent/utils/prompts.py", line 135, in parse_action_response return action_from_dict(action_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/opendevin/action/init.py", line 26, in action_from_dict raise KeyError(f"'action' key is not found in {action=}") KeyError: "'action' key is not found in action={'thought': 'How should I structure my calculator program? Should I start with a main function and define functions for each feature, or should I start with a class and implement inheritance?'}"

OBSERVATION: "'action' key is not found in action={'thought': 'How should I structure my calculator program? Should I start with a main function and define functions for each feature, or should I start with a class and implement inheritance?'}"

============== STEP 2

PLAN: write a simple calculator in python

ERROR: "'action' key is not found in action={'thought': 'How should I structure my calculator program? Should I start with a main function and define functions for each feature, or should I start with a class and implement inheritance?'}" Traceback (most recent call last): File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/opendevin/controller/agent_controller.py", line 113, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/agenthub/monologue_agent/agent.py", line 166, in step action = prompts.parse_action_response(action_resp) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/agenthub/monologue_agent/utils/prompts.py", line 135, in parse_action_response return action_from_dict(action_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/opendevin/action/init.py", line 26, in action_from_dict raise KeyError(f"'action' key is not found in {action=}") KeyError: "'action' key is not found in action={'thought': 'How should I structure my calculator program? Should I start with a main function and define functions for each feature, or should I start with a class and implement inheritance?'}"

OBSERVATION: "'action' key is not found in action={'thought': 'How should I structure my calculator program? Should I start with a main function and define functions for each feature, or should I start with a class and implement inheritance?'}" Error sending data to client INFO: connection closed Error sending data to client Cannot call "send" once a close message has been sent.

============== STEP 3

PLAN: write a simple calculator in python

ERROR: AgentThinkAction.init() missing 1 required positional argument: 'thought' Traceback (most recent call last): File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/opendevin/controller/agent_controller.py", line 113, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/agenthub/monologue_agent/agent.py", line 166, in step action = prompts.parse_action_response(action_resp) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/agenthub/monologue_agent/utils/prompts.py", line 135, in parse_action_response return action_from_dict(action_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/f.ribaldi/progetti/OpenDevin/opendevin/action/init.py", line 31, in action_from_dict return action_class(**args) ^^^^^^^^^^^^^^^^^^^^ TypeError: AgentThinkAction.init() missing 1 required positional argument: 'thought'

OBSERVATION: AgentThinkAction.init() missing 1 required positional argument: 'thought' Error sending data to client Cannot call "send" once a close message has been sent.

============== STEP 4

PLAN: write a simple calculator in python

ACTION: AgentThinkAction(thought='How should I structure my calculator program?', action='think') Error sending data to client Cannot call "send" once a close message has been sent.

============== STEP 5

PLAN: write a simple calculator in python

ACTION: AgentThinkAction(thought='How should I structure my calculator program?', action='think') Error sending data to client Cannot call "send" once a close message has been sent.

FRibaldiReply avatar Apr 04 '24 15:04 FRibaldiReply

@FRibaldiReply llama2 does not really perform well in OpenDevin because it does not have a lot of coding training, I recommend switching to dolphin-mistral:7b-v2.8-q8_0 , starcoder2 or something similar if your GPU can support it. Dolphin is specifically trained on coding datasets and it works much better in my experience.

Also, just a general note: Local llms are going to perform significantly worse than chatgpt, claude, etc. They have significantly fewer parameters. Unless you can run mixtral 8x7b locally, I imagine that would give pretty good results, but you would need 2x 3090s minimum to do so.

JayQuimby avatar Apr 04 '24 18:04 JayQuimby

We've updated the docs here, and fixed the model list. I think this one should be covered now

rbren avatar Apr 09 '24 19:04 rbren

We've updated the docs here, and fixed the model list. I think this one should be covered now

image

the models list is still wrong. that is not the model list i have loaded in ollama locally. that's just a random list of models that ollama has available.

if the config.toml specifies ollama with local model, the only model available should be the one from ollama that i am currently running.

katmai avatar Apr 10 '24 19:04 katmai

also in the settings window the model name is just displayed there from config.toml. it's not actually used for anything.

katmai avatar Apr 10 '24 20:04 katmai

the models list is still wrong. that is not the model list i have loaded in ollama locally. that's just a random list of models that ollama has available. if the config.toml specifies ollama with local model, the only model available should be the one from ollama that i am currently running.

Let me make sure I understand what you mean. I think the idea of this dropdown to let you choose another model if supported, test it out for the session, see how it works, regardless what you set as your usual model.

Do you mean that since they're local models, you need them installed, otherwise you'll never be able to choose another successfully? I mean, that obviously makes sense. It's a big list displayed all the time, even though at any given time you may need or be able to run only some. I don't know, do you want to open a new issue with a proposal of how a more useful list could look like?

For proprietary models, people usually do want to see all from that provider, since they probably have access to them.

enyst avatar Apr 10 '24 21:04 enyst

the models list is still wrong. that is not the model list i have loaded in ollama locally. that's just a random list of models that ollama has available. if the config.toml specifies ollama with local model, the only model available should be the one from ollama that i am currently running.

Let me make sure I understand what you mean. I think the idea of this dropdown to let you choose another model if supported, test it out for the session, see how it works, regardless what you set as your usual model.

Do you mean that since they're local models, you need them installed, otherwise you'll never be able to choose another successfully? I mean, that obviously makes sense. It's a big list displayed all the time, even though at any given time you may need or be able to run only some. I don't know, do you want to open a new issue with a proposal of how a more useful list could look like?

For proprietary models, people usually do want to see all from that provider, since they probably have access to them.

i think a more simple way of doing that would be to split that dropdown into "local" and "remote" models by adding another dropdown which would allow you to select between them.

however, i don't understand the logic of that settings page to begin with, since you already configured those things in 'config.toml', so it should just respect that.

katmai avatar Apr 10 '24 21:04 katmai