bug: ollama URL for chat is hardcoded to localhost:11434

[vbehar]

Version

v1.14.0

Describe the bug

when enabling the experimental feature to use a model hosted on ollama, cody can only talks to localhost:11434 - we can't configure a different URL - as we do for the completion feature for example.

Expected behavior

be able to configure the Ollama URL - same as what is currently done for the completion feature

Additional context

we have a shared ollama instance, and we would like to use it to chat, instead of asking every dev to install ollama locally

[jdorfman]

Thanks @vbehar I notified the team =)

[harshal-cuminai]

@jdorfman any timeline on when will this be fixed. I need it urgently. I can use a pre-release version on vscode.

I am currently on (pre-release) v1.17.1714749264

[jdorfman]

@harshal-cuminai I will ask them and follow up when I hear back.

[vijayabhaskar-ev]

@jdorfman Can i take up this issue?

[jdorfman]

@vijayabhaskar-ev let me check with @dominiccooney

[jdorfman]

@vijayabhaskar-ev it's yours :)

[abeatrix]

@vijayabhaskar-ev we have updated the Ollama chat client where the endpoint can now be configured with the cody.dev.models field in your VS Code settings:

	"cody.dev.models": [
		{
			"provider": "ollama",
			"model": "llama3:instruct",
			"apiEndpoint": "http://YOU_OLLAMA-HOST"
		}
	],

I did find a bug that is causing Ollama clients not to work atm, but once this PR https://github.com/sourcegraph/cody/pull/4564 is merged, we will release a patch that you can try and confirm if this issue has been resolved 🙇‍♀️

[jdorfman]

@vijayabhaskar-ev please give us feedback. Good or bad.

[vijayabhaskar-ev]

@jdorfman Using the below config is not working .

"cody.dev.models": [
		{
			"provider": "ollama",
			"model": "llama3:instruct",
			"apiEndpoint": "http://YOU_OLLAMA-HOST"
		}
	],

Currently this is the one working.

	  "cody.autocomplete.advanced.provider": "experimental-ollama",
    "cody.autocomplete.experimental.ollamaOptions": {
      "url": "http://localhost:11434",
      "model": "codellama"
    },
    

Am i missing something. Please let me know if i need to make any other config changes @abeatrix  

[ByerRA]

I can also confirm that currently using the suggested change of "cody.dev.models" in the settings file isn't working using Cody pre-release v.1.23.1718550982.

On the machine where we have Ollama running, it works just fine using the suggested "cody.dev.mnodels" configuration and I can see and select any of the Ollama models loaded to chat with.

But on another machine on the same network (like my laptop) if I use the suggested "cody.dev.models" configuration I can now at least see and select the Ollama model I configured in the "cody.dev.models" and select it for chat but when I try to chat with it, it just sits there doing nothing and I can confirm that on the machine running Ollama it's not doing anything.

[jdorfman]

@abeatrix any ideas?

cc @kynlos

[abeatrix]

@ByerRA That looks like a networking configuration issue with your Ollama instance since you are able to chat with Ollama using "cody.dev. models" when it's hosted on the same machine, but it doesn't work if the Ollama instance is hosted on a different machine?

Does it work if you try to send this curl command from your local machine to Ollama that is hosted on a separate machine?

curl http://YOU_OLLAMA-HOST/api/generate -d '{
  "model": "llama3:instruct",
  "prompt": "Why is the sky blue?"
}'

[vijayabhaskar-ev]

@ByerRA I too hosted ollama on aws and it works fine.

[ByerRA]

On any machine on my network including my laptop in question when I enter...

curl http://10.1.1.23:11434/api/generate -d '{"model": "llama3","prompt": "Why is the sky blue?"}'

I get output...

{"model":"llama3","created_at":"2024-06-19T21:24:55.226312671Z","response":"A","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:55.446845782Z","response":" classic","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:55.649150723Z","response":" question","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:55.847251318Z","response":"!\n\n","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:56.047841372Z","response":"The","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:56.249757Z","response":" sky","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:56.451015883Z","response":" appears","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:56.646876362Z","response":" blue","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:56.845326247Z","response":" because","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:57.051867034Z","response":" of","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:57.259307694Z","response":" a","done":false}
{"model":"llama3","created_at":"2024-06-19T21:24:57.476004467Z","response":" phenomenon","done":false}

And I've even tested this across my VPN and had no problems.

[abeatrix]

@ByerRA can you share your cody.dev.models settings with me so we can investigate further?

[ByerRA]

Here is my complete config file...

{
    "editor.inlineSuggest.suppressSuggestions": false,
    "workbench.startupEditor": "none",
    "cody.autocomplete.advanced.provider": "experimental-ollama",
    "cody.autocomplete.advanced.serverEndpoint": "http://10.1.1.23:11434",
    "cody.autocomplete.experimental.ollamaOptions": {
        "url": "http://10.1.1.23:11434",
        "model": "deepseek"
    },
    "cody.dev.models": [{
        "provider": "ollama",
        "model": "deepseek",
        "apiEndpoint": "http://10.1.1.23:11434",
    }],
    "editor.minimap.enabled": false,
    "cody.experimental.ollamaChat": true,
    "redhat.telemetry.enabled": false,
    "cody.autocomplete.languages": {
        "*": true
    },
    "extensions.ignoreRecommendations": true,
    "terminal.integrated.enableMultiLinePasteWarning": "never",
    "explorer.confirmDelete": false
}

[abeatrix]

@ByerRA thank you! Is deepseek the model name you pulled from Ollama? Or should it be deepseek-coder?
I'm trying to reproduce the issue but couldn't find a model with just deepseek on Ollama library 😅

[ByerRA]

Yes, you are correct, it should be "deepseek-coder", my typing mistake.

[abeatrix]

@ByerRA oh nice! Did updating the model name fix the issue for you?

[valerybugakov]

The cody.dev.models setting solves the original issue. Feel free to reopen if it's not the case.