Replace openai to llama2?

[hackhy]

How can I replace the OpenAI API with Llama2?

[joonspk-research]

There is a no built-in function for that yet, but it should be relatively simple to make it happen. The main place you will need to edit would be: generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py

This is where all the actual API calls to OpenAI are made. You will want to edit the functions there to instead call Llama2. (You may have to edit utils.py + a few prompts if they break when using it with Llama)

[huvers]

llama.cpp has a neat api_like_OAI.py drop in Flask server to mimic all the OpenAI API calls. You should be able to run that, then set ``openai.api_base = `"http://127.0.0.1:8081"```, and llama2 should work. I'll hopefully have time to try this weekend.

[wonhyeongseo]

May I please try writing up a PR using @huvers 's method? This thread seems to be a part of https://github.com/joonspk-research/generative_agents/issues/3 Kudos for opensourcing this wonderful experiment!

[AWAS666]

llama.cpp has a neat api_like_OAI.py drop in Flask server to mimic all the OpenAI API calls. You should be able to run that, then set ``openai.api_base = `"http://127.0.0.1:8081"```, and llama2 should work. I'll hopefully have time to try this weekend.

Yeah that works but you can also use textgenwebui with the openai extension, which is likely what many playing with llama based llm have installed already anyway.

[SaturnCassini]

This version runs on GPT4All model which is free and doesn't require a GPU

https://github.com/SaturnCassini/gpt4all_generative_agents

[pjq]

Probably you need this, and replace the base URL with your local URL.

• https://github.com/abetlen/llama-cpp-python

pip install llama-cpp-python[server]
python3 -m llama_cpp.server --model models/7B/ggml-model.bin

The update to the baseUrl

• http://localhost:8000/

[ishaan-jaff]

@hackhy I'm the maintainer of liteLLM https://github.com/BerriAI/litellm/ - we make it easy to switch between models.

Here's how you can call llama2 using liteLLM

from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# llama2 call
response = completion(model="meta-llama/Llama-2-7b-hf", messages=messages)

[PiPi-happy]

"Excuse me, if I want to replace it with Azure OpenAI, what should I do?"

[comaniac]

@joonspk-research I have the same requirement and have changed the code in my fork. Would you want me to file a PR or it's already an ongoing feature on your side? I could send out a PR for it early next week if needed. Thanks.

[certainforest]

@comaniac how are you handling inconsistent completion structure? I'm running into issues with implementing llama because prompt responses don't appear to be coming back in json format and/or are inappropriate. 😓

[comaniac]

I actually commented this to another issue. I did encounter this problem and my feeling is this framework is tightly-coupled with certain OpenAI models in terms of prompts and response formats. We may need some efforts (e.g., prompt engineering) to make other models work seamlessly. Meanwhile, the first step toward to this goal is to refactor the framework so that we can configure the model. Then we could start tweaking.

[ElliottDyson]

I actually commented this to another issue. I did encounter this problem and my feeling is this framework is tightly-coupled with certain OpenAI models in terms of prompts and response formats. We may need some efforts (e.g., prompt engineering) to make other models work seamlessly. Meanwhile, the first step toward to this goal is to refactor the framework so that we can configure the model. Then we could start tweaking.

@metrics-dawg @comaniac We're working on this over at OSGA (open source generative agents: https://github.com/ElliottDyson/OSGA), you're welcome to join us, there's a bit more activity on the Discord, but I've made good progress to getting working outputs out of llama 2.

[babytdream]

I use my own api_base. Whenever run xxx ends, this error will appear. Has anyone encountered this problem?

-==- -==- -==- 
Traceback (most recent call last):
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 471, in open_server
    rs.start_server(int_count)
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 379, in start_server
    next_tile, pronunciatio, description = persona.move(
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 222, in move
    plan = self.plan(maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 148, in plan
    return plan(self, maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 959, in plan
    _determine_action(persona, maze)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 573, in _determine_action
    generate_task_decomp(persona, act_desp, act_dura))
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 164, in generate_task_decomp
    return run_gpt_prompt_task_decomp(persona, task, duration)[0]
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 439, in run_gpt_prompt_task_decomp
    output = safe_generate_response(prompt, gpt_param, 5, get_fail_safe(),
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py", line 262, in safe_generate_response
    return func_clean_up(curr_gpt_response, prompt=prompt)
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 378, in __func_clean_up
    duration = int(k[1].split(",")[0].strip())
IndexError: list index out of range
Error.

[ElliottDyson]

I use my own api_base. Whenever run xxx ends, this error will appear. Has anyone encountered this problem?

-==- -==- -==- 
Traceback (most recent call last):
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 471, in open_server
    rs.start_server(int_count)
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 379, in start_server
    next_tile, pronunciatio, description = persona.move(
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 222, in move
    plan = self.plan(maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 148, in plan
    return plan(self, maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 959, in plan
    _determine_action(persona, maze)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 573, in _determine_action
    generate_task_decomp(persona, act_desp, act_dura))
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 164, in generate_task_decomp
    return run_gpt_prompt_task_decomp(persona, task, duration)[0]
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 439, in run_gpt_prompt_task_decomp
    output = safe_generate_response(prompt, gpt_param, 5, get_fail_safe(),
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py", line 262, in safe_generate_response
    return func_clean_up(curr_gpt_response, prompt=prompt)
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 378, in __func_clean_up
    duration = int(k[1].split(",")[0].strip())
IndexError: list index out of range
Error.

Yep, I encountered it too. Unfortunately can't remember what the fix was, but you're welcome to head over to the OSGA GitHub and use the code from there (which includes the fix, as well as other fixes)

[Murat2283plus]

"Excuse me, if I want to replace it with Azure OpenAI, what should I do?"“请问，如果我想更换为 Azure OpenAI，该怎么办？”

did you success?

[ketsapiwiq]

Here's a draft that kinda works: https://github.com/joonspk-research/generative_agents/pull/155/files