[FEAT]: Add bearer token to header for Ollama LLM Provider and UI for setting this.

[sheneman]

What would you like to see?

This should be really simple, high-impact, low-hanging fruit for the Desktop and Web App!!

Please add an optional advanced setting for Ollama providers where you can specify a bearer token.

Currently, AnythingLLM does not appear to have a mechanism or user interface for setting the bearer token in the http header for Ollama LLM Providers (Ollama endpoints). Other tools like WebUI and Enchanted provide this, which is really helpful if your central Ollama infrastructure supports Ollama API keys in this way using an authenticating proxy like NGINX, Caddy, etc.

This tiny missing feature is the only thing preventing much larger adoption of AnythingLLM within our organization.

Thank you so much for your consideration!

[ssbodapati]

This works for me, @timothycarambat and @sheneman. Since the genericLLM constructor is using openAI constructor, the API key passed/set as env variable will be used for the header construction.

the below settings worked for me,

Base url : {host}/v1/ [ activating Open AI completions, reference.] API Key : {token} [ Just the token, "Bearer " is appended by the open AI SDK ] Chat Model Name: llava ollama log -> [GIN] 200 | 57.044524133s | POST "/v1/chat/completions"

[timothycarambat]

Where BaseURL is the URL of the Ollama instance, correct?

[sheneman]

Yes, @timothycarambat and @ssbodapati -- for client use cases that are expecting an OpenAI Chat Completion endpoint, that would work. But I am interested in supporting a bearer token that can be optionally passed via the header for Ollama native endpoints. Some clients do not support OpenAI style endpoints.

The popular Enchanted iOS app is one example.

Here is a relevant article on how to configure an auth proxy for Ollama.

https://www.arsturn.com/blog/understanding-authentication-in-ollama

And here is a relevant repo:

https://github.com/bartolli/ollama-bearer-auth

We are interested in having AnythingLLM support bearer tokens for native Ollama completion endpoints. Among other things, that would allow a more dynamic selection of different models, etc via pulldown menu in Workspace settings, etc. and be more similar to other tools (like Enhanted) that support auth to Ollama via bearer tokens.

[ssbodapati]

@timothycarambat yes, the base url is Ollama host url I made a quick draft, here is the PR -> 2749 ollama client auth token #3005

@sheneman I'm able to fetch model list through /api/tags on a secure Ollama server by setting a bearer token.This should work for all other APIs as well.

Ollama LLM Preferences with Auth Token