feature: add llama api using local models

[mkellerman]

This allows the user to un-comment the section in the docker-compose.yml file and .env to either use the mocked-api or the llama-api.

The user can then load local models, and use chatgpt-web against it.

[Niek]

Thanks for your contribution!

I'm a bit in doubt which approach to take: your PR or what @mudler proposed in https://github.com/Niek/chatgpt-web/issues/105 - I'm leaning toward his proposal because it's compatible with more models, and since the API is hosted in a separate repo/docker image it should be better updated.

Any thoughts?

[Niek]

I see that you actually worked in this in https://github.com/mkellerman/chatgpt-web/tree/feature/add-llama-cli, great :)

[mkellerman]

Yes, weither approaches would work. I dont see a negative impact to have both repo document/showcase one another. And it allows both projects to grow and thrive.

[mkellerman]

Having it hear helps with the development work of the UI. And having it overthere helps people working on their MLs.

[Niek]

OK, I will support both - but first I will contribute a Dockerfile to the llama-cpp-python repo, so we can just pull the latest version and not maintain that in this repo.

Edit: done, see https://github.com/abetlen/llama-cpp-python/pull/73

[countzero]

@mkellerman & @Niek I am not sure, if I am missing anything, but isn't this capability already implemented by https://github.com/Niek/chatgpt-web/pull/494?

You can simply start a local OpenAI API compatible llama-server hosting a local model and point chatgpt-web against it:

[Niek]

Yes, this is an old issue that can be closed now. PRs are welcome with some docs around this though :)