Support for opensource models like LLama

[pradeepdev-1995]

Is it possible to configure opensource models like dolly,llama,,etc instead of openai models in simpleaichat and do prompting also.

[Vokturz]

I've created a PR # 52 which allows to use a custom API URL endpoint. For llama models, you can start the llama.cpp python web-server and then change the api_url of AIChat:

ai = AIChat(api_key='None', api_url='http://localhost:8000/v1/chat/completions', console=False)

[minimaxir]

I merged #52 since that is a fair fix for a bug, but I am uncertain on how high of a priority to develop for open source models like llama.ccp is, particularly since they may have different APIs that are unique and don't play nice with each other.

It is definitely within scope and on the roadmap, though.

[araichev]

Does this PR work with (local) GPT4All models too?

[Vokturz]

Does this PR work with (local) GPT4All models too?

I don't think so, GPT4All API server does not have an implementation for the chat/completions URL.

[Xoeseko]

Does this PR work with (local) GPT4All models too?

I don't think so, GPT4All API server does not have an implementation for the chat/completions URL.

Actually, in the link you sent, L49 gives the completions endpoint which seems to be compatible. All routes refering to that router are subroutes of the chat route.

I also got the chance to test and it seems that the GPT4All API server is compatible. However, I needed to find a way to create a session without passing an api key which led to this PR #62. @Vokturz Were you able to use other models without this implementation ?

[Vokturz]

@Xoeseko but if you take a closer look at chat_completion function, you will notice it is currently just a structural placeholderwith no actual implementation. I've tried following the instructions from GPT4ALL-API README with no success.