Add support for tabAutocompleteModel remotely

[xndpxs]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [x] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:EndeavourOS
- Continue:0.9.123
- IDE:VSCodium

Description

Hi all, my Llama3 is working flawlessly on LAN. The problem is that I need starcoder2 for the tabAutoCompleteModel Option. I can see both models with ollama list:

ollama list
NAME                    ID              SIZE        MODIFIED     
llama3:latest           a6990ed6be41    4.7 GB      5 hours ago
starcoder2:latest       f67ae0f64584    1.7 GB      2 hours ago

Continue is working with llama3 remotely, but I can't seem to be able to configure tabAutoCompleteModel in Continue with starcoder2 from the same server, I am getting this error: The model 'starcoder2' was not found But In fact as you can see it is running.

To reproduce

I am configuring

  "tabAutocompleteModel": {
    "title": "Starcoder2",
    "provider": "ollama",    
    "model": "starcoder2",
    "apiBase": "http://ip:port"

Log output

http://ip:port/api/generate The model "starcoder2" was not found. To download it, run `ollama run starcoder2`.

[sestinj]

@xndpxs this is a consequence of how Ollama manages models—they won't always acknowledge an alias to the same one you have downloaded. If you set "model": "starcoder2:latest" in your config instead, I believe this could solve the problem. Let me know if not!

[xndpxs]

Nope didn't work. I've tried with all combinations possible (latest, 2b, -, :) none of them worked. I've tried also specifying the api and not specifying it. None of that worked. I tried with this

 "tabAutocompleteModel": {
    "title": "Tab Autocomplete Model",
    "provider": "ollama",
    "model": "llama3",
    "apiBase": "http://ip:port"
  },

And it doesn't give me errors, but it looks like it is using llama3 instead of starcoder. Maybe it can't be seen by continue, or it can't be served correctly? because llama3 is occupying already the port?

[ahoplock]

Not sure if it's related but I had pretty much the same issue yesterday when I installed the latest version (0.8.25) of the VSCode* version of the extension. While chat worked just fine with my "remote" Ollama install on a local network, auto-complete was logging errors in the VSCode debug console along the lines of, Error generating autocompletion: Error: Failed to connect to local Ollama instance. I rolled back a couple versions until I found one that worked (0.8.23) which was from 3 weeks ago. After rolling back, I was able to use the Starcoder2 3B model for auto-complete with a config like the following,

  "tabAutocompleteModel": {
    "title": "starcoder2:3b",
    "model": "starcoder2:3b-q6_K",
    "apiBase": "http://ip:port",
    "provider": "ollama"
  },

*not that I think it matters but I actually use VSCodium

[jarenal]

I don't know if this could be relevant to anyone, but in my case it works correctly with this setup:

  "tabAutocompleteModel": {
    "title": "Starcoder2 3b",
    "provider": "ollama",
    "model": "starcoder2:3b",
    "apiBase": "http://x.x.x.x:11434"
  }

My IDE is PHPStorm and I use Ollama installed on another computer of my local network.

When I execute ollama list I got:

llama3:latest                   a6990ed6be41    4.7 GB  3 days ago
starcoder2:3b                   f67ae0f64584    1.7 GB  2 days ago

[xndpxs]

It was fixed! It looks like an environment variable problem. when I run as this:

OLLAMA_HOST=ip:port ollama list

NAME                    ID              SIZE        MODIFIED     
llama3:latest           a6990ed6be41    4.7 GB      5 hours ago

It just shows 1 of the 2 models I have installed It looks like the model wasn't installed in the env variable, if that makes any sense? Then I installed with

OLLAMA_HOST=ip:port ollama pull starcoder2:latest

and then It woked good. Honestly, I don't remember if I installed starcoder2 before or after I modified the systemd config file:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
Environment=OLLAMA_HOST=ip:port
User=ollama
Group=ollama
Restart=always
RestartSec=3

[Install]
WantedBy=default.target

[XinLiShen]

I've figured it out. We got the same problem. The version I downloaded before was starcoder2:latest, but it would prompt that it couldn't be detected, even though it is actually starcoder2:3b. You can first not delete it, but instead directly run ollama run starcoder2:3b, and it will be downloaded immediately, and then you can delete the latest version, and that should work.

[XinLiShen]

It was fixed! It looks like an environment variable problem. when I run as this:

OLLAMA_HOST=ip:port ollama list

NAME                    ID              SIZE        MODIFIED     
llama3:latest           a6990ed6be41    4.7 GB      5 hours ago

It just shows 1 of the 2 models I have installed It looks like the model wasn't installed in the env variable, if that makes any sense? Then I installed with

OLLAMA_HOST=ip:port ollama pull starcoder2:latest

and then It woked good. Honestly, I don't remember if I installed starcoder2 before or after I modified the systemd config file:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
Environment=OLLAMA_HOST=ip:port
User=ollama
Group=ollama
Restart=always
RestartSec=3

[Install]
WantedBy=default.target

We got the similar problem. The version I downloaded before was starcoder2:latest, but it would prompt that it couldn't be detected, even though it is actually starcoder2:3b. You can first not delete it, but instead directly run ollama run starcoder2:3b, and it will be downloaded immediately, and then you can delete the latest version, and that should work.