interop with oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#examples

[disarticulate]

text-generation-webui appears to have an OpenAI compatible API. It's not clear when you have the openAI docs how to use it that way. Looks like there's a custom API option, but seems excessive if we're just talking about inserting an endpoint described by an openAPI.

[yuhongsun96]

I'm sorry, I'm not sure I understand what you're asking. Could you clarify, thanks?

[maciu665]

Danswer is reportedly able to use custom models compatible with OpenAI, such as text-generation-webui. Is it really possible? Are there any examples?

After setting everything according to https://docs.danswer.dev/gen_ai_configs/custom_server I get HTTPConnectionPool(host='host.docker.internal', port=11434): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f98ec79e810>: Failed to establish a new connection: [Errno -2] Name or service not known')) while calling the text-generation-webui from the terminal gives a correct response.

[disarticulate]

Yes, we want to use a custom but OpenAI compatible API endpoint.

it's not clear how to do that.

On Tue, Jan 30, 2024, 03:39 maciu665 @.***> wrote:

Danswer is reportedly able to use custom models compatible with OpenAI, such as text-generation-webui. Is it really possible? Are there any examples?

After setting everything according to https://docs.danswer.dev/gen_ai_configs/custom_server I get HTTPConnectionPool(host='host.docker.internal', port=11434): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f98ec79e810>: Failed to establish a new connection: [Errno -2] Name or service not known')) while calling the text-generation-webui from the terminal gives a correct response.

— Reply to this email directly, view it on GitHub https://github.com/danswer-ai/danswer/issues/826#issuecomment-1916437768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEFHWPILPKBV3P6PV6QN27TYRC5VZAVCNFSM6AAAAABAOXBYA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJWGQZTONZWHA . You are receiving this because you authored the thread.Message ID: @.***>

[todaywasawesome]

@disarticulate To use danswer with text-generation-webui just set the environment variable for the endpoint.

GEN_AI_API_ENDPOINT=http://[URL:port]/v1/

In text-geneeration-webui make sure the API is enabled and you built with openai. https://github.com/oobabooga/text-generation-webui/wiki/06-%E2%80%90-Session-Tab https://github.com/oobabooga/text-generation-webui/wiki/09-%E2%80%90-Docker