[BUG] - Local Model LLM works, but not Embedding

[ajweber]

Description

If you download a gguf model and update the LLM URL settings to the proper port where kotaemon is loading the model, testing against the "ollama" LLM works.

However, the Embedding does not work this way. If you are able to d/l and serve the local model (LLM), shouldn't you be also allowing for the Embedding to be a local model as well?

Workaround would appear to be to separately download and install ollama complete setup. I was hoping to continue to work "within kotaemon" instead of laying external pieces.

Reproduction steps

Choose Embeddings Tab.
Select ollama.
Optional: Change port to 31415 (where kotaemon loads and exposes LOCAL_MODEL).
Save.
Click Test.

Fails.

Screenshots

No response

Logs

- Testing model: ollama
- Sending a message `Hi`
- Connection failed. Got error: RetryError[]

Browsers

Microsoft Edge

OS

Linux

Additional information

LLM test with this port works. Model on that config is the full path to the local gguf. Returns: "- Connection success. Got response: How can I assist you today?"

Testing model from huggingface: Llama-3.1-Storm-8B-Q6_K_L.gguf

[linuaries]

Should configure as following Resourcess->Index Collections-> File(FileIndex):

embedding: ollama
max_file_size: 1000
max_number_of_files: 0
private: false
supported_file_types: .png, .jpeg, .jpg, .tiff, .tif, .pdf, .xls, .xlsx, .doc, .docx,
  .pptx, .csv, .html, .mhtml, .txt, .zip

[ajweber]

That is already configured as you suggested.

[JaswanthMannem]

same issue. Did anyone get it right.

[Sivan22]

for me it worked as expected. with this embeddings model: https://huggingface.co/Ralriki/multilingual-e5-large-instruct-GGUF

[DangerousBerries]

I'm also getting - Connection failed. Got error: RetryError[] im using docker and ollama is in docker too

[taprosoft]

To use Ollama in Docker, you might want to change the base URL of the Ollama server https://stackoverflow.com/questions/31324981/how-to-access-host-port-from-docker-container

[DangerousBerries]

To use Ollama in Docker, you might want to change the base URL of the Ollama server https://stackoverflow.com/questions/31324981/how-to-access-host-port-from-docker-container

I tried to use the base URL that works for me in AnythingLLM but it still didn't work in kotaemon. By the way is there a way to change the main LLM's system prompt in Docker?