llama2 dont work correctly, what is the right model of ollama to use with open devin?

[Jmzp]

Describe the bug

Steps to Reproduce

1. Use llama2 as local model

WORKSPACE_DIR="./workspace"
LLM_API_KEY="ollama"
LLM_BASE_URL="http://localhost:11434"
LLM_EMBEDDING_MODEL="local"
LLM_MODEL ='ollama/llama2'

Actual behavior Dont work

[footedesign]

https://github.com/OpenDevin/OpenDevin/issues/311#issuecomment-2026592291

[Jmzp]

Ollama responds, but OpenDevin can't interpret the responses correctly

[AnoshkaAlexandr]

I have the same issue on macOS 14.4.1 (23E224) M1 Max

LLM_API_KEY="11111111111111" LLM_MODEL="ollama/llama2" WORKSPACE_DIR="./workspace" LLM_BASE_URL="http://localhost:11434" LLM_EMBEDDING_MODEL="llama2"

when run ollama serve

STEP 0

    PLAN:

    🔵 0 simple site

Repaired JSON: {"action": "think", "args": {"thought": "Hmm, I think I should start by running ls to see what files are in this directory."}} Original JSON: { "action": "think", "args": { "thought": "Hmm, I think I should start by running ls to see what files are in this directory." } }

    ACTION:
    AgentThinkAction(thought='Hmm, I think I should start by running `ls` to see what files are in this directory.', action='think')

============== STEP 1

    PLAN:

    🔵 0 simple site

Repaired JSON: "" Original JSON:

    AGENT ERROR:
    'str' object has no attribute 'copy'

Traceback (most recent call last): File "/Users/alex/Documents/Projects/python/devin/OpenDevin/opendevin/controller/agent_controller.py", line 89, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex/Documents/Projects/python/devin/OpenDevin/agenthub/monologue_agent/agent.py", line 164, in step action = prompts.parse_action_response(action_resp) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex/Documents/Projects/python/devin/OpenDevin/agenthub/monologue_agent/utils/prompts.py", line 135, in parse_action_response return action_from_dict(action_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex/Documents/Projects/python/devin/OpenDevin/opendevin/action/init.py", line 24, in action_from_dict action = action.copy() ^^^^^^^^^^^ AttributeError: 'str' object has no attribute 'copy'

    OBSERVATION:
    'str' object has no attribute 'copy'

---

and another error like this when ollama running in background

(.venv) (base) alex@MacBook-Pro-Alex OpenDevin % PYTHONPATH=pwd python opendevin/main.py -d ./workspace/ -i 100 -t "Write a bash script that prints 'hello world'" Running agent MonologueAgent (model: ollama/llama2, directory: ./workspace/) with task: "Write a bash script that prints 'hello world'"

============== STEP 0

    PLAN:

    🔵 0 Write a bash script that prints 'hello world'

Repaired JSON: {"action": "think", "args": {"thought": "I should start by making a plan for my task. I want to break it down into smaller, manageable steps and set specific goals for each step."}} Original JSON: { "action": "think", "args": { "thought": "I should start by making a plan for my task. I want to break it down into smaller, manageable steps and set specific goals for each step." } }

    ACTION:
    AgentThinkAction(thought='I should start by making a plan for my task. I want to break it down into smaller, manageable steps and set specific goals for each step.', action='think')

============== STEP 1

    PLAN:

    🔵 0 Write a bash script that prints 'hello world'

Repaired JSON: {"thought": "I should start by making a plan for my task. I want to break it down into smaller, manageable steps and set specific goals for each step."} Original JSON: { "thought": "I should start by making a plan for my task. I want to break it down into smaller, manageable steps and set specific goals for each step." }

    AGENT ERROR:
    "'action' key is not found in action={'thought': 'I should start by making a plan for my task. I want to break it down into smaller, manageable steps and set specific goals for each step.'}"

Traceback (most recent call last): File "/Users/alex/Documents/Projects/python/devin/OpenDevin/opendevin/controller/agent_controller.py", line 89, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex/Documents/Projects/python/devin/OpenDevin/agenthub/monologue_agent/agent.py", line 164, in step action = prompts.parse_action_response(action_resp) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex/Documents/Projects/python/devin/OpenDevin/agenthub/monologue_agent/utils/prompts.py", line 135, in parse_action_response return action_from_dict(action_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex/Documents/Projects/python/devin/OpenDevin/opendevin/action/init.py", line 26, in action_from_dict raise KeyError(f"'action' key is not found in {action=}") KeyError: "'action' key is not found in action={'thought': 'I should start by making a plan for my task. I want to break it down into smaller, manageable steps and set specific goals for each step.'}"

[Jmzp]

I tried code llama mistral and OpenDevin don't work correctly. Only mistral works so so

[AnoshkaAlexandr]

Is your inner browser work correctly?

with mistral

[AnoshkaAlexandr]

yes, with mistral it work time from time but terminal, browser and planner in the workspace panel don't work

[Jmzp]

Browser and Planner is still under development, no? :thinking:

[Toasterme]

I tried code llama mistral and OpenDevin don't work correctly. Only mistral works so so

what is the config for mistral ? i cannot get it to work with oolama with TheBloke/CodeLlama-7B-Instruct-GGUF/codellama-7b-instruct.Q8_0.gguf

tried Mistral too with NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q8_0.gguf

For those, what are the LLM_MODEL + LLM_EMBEDDING_MODEL

[stratte89]

Same I try to get it to work as well with lm-studio or ollama, both arent working, im on ubuntu

for ollama LLM_BASE_URL="127.0.0.1:11434" LLM_MODEL="ollama/mistral" LLM_EMBEDDING_MODEL="local" WORKSPACE_DIR="./workspace"

i run ollama serve in one terminal and ollama run mistral in another, also the server uvicorn opendevin.server.listen:app --port 3000 and npm start. Open AI Api works fine.

Oops. Something went wrong: Error condensing thoughts: No healthy deployment available, passed model=ollama/mistral

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/stratte/Schreibtisch/AI/OpenDevin-main/opendevin/controller/agent_controller.py", line 89, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/stratte/Schreibtisch/AI/OpenDevin-main/agenthub/monologue_agent/agent.py", line 151, in step self._add_event(prev_action.to_dict()) File "/home/stratte/Schreibtisch/AI/OpenDevin-main/agenthub/monologue_agent/agent.py", line 94, in _add_event self.monologue.condense(self.llm) File "/home/stratte/Schreibtisch/AI/OpenDevin-main/agenthub/monologue_agent/utils/monologue.py", line 36, in condense raise RuntimeError(f"Error condensing thoughts: {e}") RuntimeError: Error condensing thoughts: No healthy deployment available, passed model=ollama/mistral

OBSERVATION:
Error condensing thoughts: No healthy deployment available, passed model=ollama/mistral

Exited before finishing

[johnnyaug]

Getting a similar issue with:

LLM_API_KEY=""
WORKSPACE_DIR="./workspace"
LLM_BASE_URL="http://localhost:11434"
LLM_MODEL="ollama/llama2"
LLM_EMBEDDING_MODEL="llama2"

[techriffic]

anyone found a solution? not working for me either.. LLM_API_KEY=sk-1234" WORKSPACE_DIR="./workspace" LLM_BASE_URL="http://localhost:11434" LLM_MODEL="ollama/mistral:7b-instruct-q4_K_M"

[DGdev91]

Hermes-2-Pro-Mistral-7B kinda worked for very simple tasks (example: write an hello world in python), but it gets easly confused with more complex tasks.

If anyone is willing to try that, i'm running it on Koboldcpp with this config:

LLM_API_KEY="." LLM_MODEL="gpt-4-0125-preview" LLM_BASE_URL="http://localhost:5001/v1/" WORKSPACE_DIR="./workspace"

[Toasterme]

Hermes-2-Pro-Mistral-7B kinda worked for very simple tasks (example: write an hello world in python), but it gets easly confused with more complex tasks.

If anyone is willing to try that, i'm running it on Koboldcpp with this config:

LLM_API_KEY="." LLM_MODEL="gpt-4-0125-preview" LLM_BASE_URL="http://localhost:5001/v1/" WORKSPACE_DIR="./workspace"

Tried the exact same GGUF as you under LM Studio and got : Oops. Something went wrong: OpenAIException - Error code: 400 - {'error': '<LM Studio error> Unknown exception during inferencing.. Error Data: n/a, Additional Data: n/a'}

What are the LLM_EMBEDDING_MODEL you used ? you didn't specify any ? I put Local.

[DGdev91]

Tried the exact same GGUF as you under LM Studio and got : Oops. Something went wrong: OpenAIException - Error code: 400 - {'error': '<LM Studio error> Unknown exception during inferencing.. Error Data: n/a, Additional Data: n/a'}

What are the LLM_EMBEDDING_MODEL you used ? you didn't specify any ? I put Local.

Yes, didn't specify one. But that error seems more like something wrong on lmstudio side. You should probably look there what's happening

[johnnyaug]

EDIT: sorry, wrong issue.

~This is my first time working with LiteLLM so please take this with a grain of salt, but I think the problem was introduced when we switched to using LiteLLM Router.~

~According to the docs, the model to the constructor should be passed together with the provider.~

~I think the solution should be something like I'm try to do in this PR.~

[rbren]

See https://github.com/OpenDevin/OpenDevin/blob/main/docs/documentation/LOCAL_LLM_GUIDE.md for docs on how to use llama/ollama