rengine
rengine copied to clipboard
yogeshojha
custom api
@yogeshojha related to #1114 @SubGlitch1 👍 But, please, document explicitely your changes
@yogeshojha I had seen a lot of people wanting to be able to use their own custom/local LLms. This can be done quite easily by changing the code to include an env variable which the gpt.py treats as the base url if it exists.
There are a growing number of openai compatible api server on github one of the most known ones being text-generation-ui. By using a project like this a user can host their own LLM api server which has the exact same scheme as the official openai api.
This obviously greatly increases privacy as the information of targets doesn't get sent to openai anymore. It also makes the gpt feature free as you don't have to pay for the api tokens anymore obviously. It also enables users who live in geo blocked areas to use the gpt feature.
Thanks @SubGlitch1 , do all these api have same response? Can you provide me the response for text-generation-ui
@yogeshojha text-generation-webui has the exact same scheme of openai. The request/response scheme will be exactly the same. I have tested on my own PC Test details OS: amd64 arch Linux llm: openchat3.5_Q3
@yogeshojha
https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#sse-streaming
You can see the examples here. A lot of projects which use the openai api e.g. gpt-engineer also have options to change the base api url for the exact same reason.
@SubGlitch1 I didn't know the subject, but copilot is usable with your modifications ?
@psyray the issue you are referencing has been opened because the op wants to get gpt4 access for free. His suggested integration is not possible as copilot isn't open source but by integrating my solution one could host any model that they want for free.
@psyray the issue you are referencing has been opened because the op wants to get gpt4 access for free. His suggested integration is not possible as copilot isn't open source but by integrating my solution one could host any model that they want for free.
Thanks 👍