please add local llama support

[soragui]

using openai api is a little expensive to generate ,can add support for local llama ....

[nalbion]

@soragui how does Local Llama compare to LocalAI? (see issue #92)

[deepweather]

Nice request, highly support it! Being able to run gpt pilot with local models (maybe in the future fine tuned experts for different parts in the chain) and therefore have control over their instruction tuning would be wonderful.

@nalbion

I would suggest to use LocalAI over direct Llama, because this is much easier to maintain as LocalAI implements the OpenAI API specs.

[nalbion]

Thanks @deepweather

[nalbion]

@soragui @deepweather I don't suppose you'd be able to test this for us?

This may already be supported. Apparently you just need to set OPENAI_API_BASE=http://localhost:8080 and run LocalAI.

[adsanz]

I've tested this with a basic LocalAI set-up. It may be nice to add some tips so people don't go crazy. For example, users should set a big enough context size (above 1024) to allow gpt-pilot to work properly. Also I tested this with one of the vicuna 13b models from the gallery, it bugged quite a lot and usually didn't generate useful answers. If anybody wants me to try things out with LocalAI let me know and I'll get back here with results

[LeonOstrez]

@adsanz thats amazing! Thanks so much for testing it.

It would be very helpful if you could add to readme what you had to do for setup so we make it easier for others who want to try it out 🙏

[adsanz]

I'll commit a PR but I'd like to carry out a bit more testing, at the moment I didn't get anything useful out of local LLMs, GPT4 just blows them away. Also I noticed that on local LLMs it just goes straight to the user stories, and it does not make any questions. I will mention my findings in the PR as well if you like. @LeonOstrez

[Deng-Xian-Sheng]

Why is the local area not as cloud like, but openai is really too expensive

[Deng-Xian-Sheng]

I have a good idea that Intel BigDL may have an acceleration effect on local models, as Intel is promoting the local operation of large models. For example, the large model of 13B, optimized by BigDL for the 12th generation Intel, has 20 tokens per second

[Deng-Xian-Sheng]

How to start local now, is there README?Thanks

[soragui]

@Deng-Xian-Sheng you can start with localai https://localai.io/basics/getting_started/

[nalbion]

Thanks @adsanz - hopefully some of the issues with the local & open source LLMs will be resolved by #188

[Deng-Xian-Sheng]

@Deng-Xian-Sheng you can start with localai https://localai.io/basics/getting_started/

Thank you, I think this can be added to README to make this project more popular.

[nalbion]

@soragui @Deng-Xian-Sheng as per my comments in #188 I think we might have more luck using other models if we update all of the prompts to include a JSON schema, otherwise the expected response format is too open to interpretation by other models.

[dalamar96]

I really appreciate the replies I have gotten and the thread has received as well. I was able to get LocalAI going with Llama2 13B GGUF, Seems I am having trouble with the postgresql user_app table with the app-id, which doesn't exist, so still working through that, I look forward to working it out and maybe getting this thing to really do some fun stuff! :)

[Gitterman69]

have you guys tried llm studio as ooenai replacement??

[benrman]

I keep getting this error. I changed the .env like in the other examples but mine won't do it. IDK

[e-p-s]

any news here?