Set custom ollama endpoint

[matthieusieben]

Ollama is running on another machine on my network.

I keep getting the following error: "Failed to connect to Ollama server http://127.0.0.1:11434. Please start Ollama server or check ollama endpoint."

But I can't figure from the docs where to set the "ollama endpoint" that I need to check.

[onkanat]

https://code.visualstudio.com/docs/intelligentapps/models

To reference a self-hosted model or locally-running Ollama model:

Select + Add model in the model catalog

Choose between Ollama or a custom model in the model Quick Pick

Provide details to add the model

[a1exwang]

Hi @onkanat, after you select Provide details to add the model here

You can input a custom Ollama endpoint that points to your other machine. The format is something like http://<other machine's IP address>:11434/

[matthieusieben]

I know that models can be added manually through "Add Model" > "Add Ollama Model" > "Provide custom Ollama endpoint".

It just seems like, from the message error "Please start Ollama server or check ollama endpoint", it would be possible to set the ollama endpoint in some config somewhere and be able to use the remote ollama server as if is was locally installed.

My question is thus, does this config exist indeed? And if so, how to set it up ?

[a1exwang]

@matthieusieben Oh I understand. There is a config file storing the Ollama endpoint and it is in %userprofile%\.aitk\models\my-models.yml and you can search for Ollama in it and update the endpoint. After you change it, make sure to reload vscode

[matthieusieben]

I already have that setup.

I guess the confusion comes from the fact that the "Ollama Models" section in the Catalog page:

1. Does not list all the Ollama models defined in my ~/.aitk/models/my-models.yml
2. Those listed, and installed, are marked as "✓ Added (1)" (see screenshot)
3. Pressing "+ Add" causes the error mentioned (it's not using the right host to perform the ollama pull operation)

Just to clarify: This is not blocking for me in anyway. This is just, IMHO, a "gap" in this product's UX.

Feel free to close.

[a1exwang]

Thanks for the clarification. Now it is clear.

1. "Ollama Models" section in the Catalog page is some predefined well tested Ollama models so cannot be adjusted by users.
2. These models are lacking UX to choose the endpoint. The workaround is to edit the yaml manually or to add custom ollama models.

I think there is definitely some improvements we can do, like allowing users to choose endpoints (defaults to localhost:11434) when adding Ollama models from model catalog