Using llama3 for causes never ending response.

[adamzenith]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Ubunutu 22.04
- Continue:0.8.24
- IDE: VScode 1.88.1
- Model: Llama 3 8b:instruct

Description

I think the llama3 version that Ollama uses has a different stop string than continue is expecting. As seen in the screenshot, it outputs an <|eot_id|>, but then continues. I have seen this go on until no more token can be generated.

To reproduce

use llama3 8b as a chat model and ask it anything.

Log output

No response

[evertjr]

I switched to the preview release and it works fine in the chat area now. But I couldn't get the base model to work for tab autocomplete though, so I'm using codeqwen for now.

[sestinj]

As mentioned this should be solved in pre-release, but if you are still having the problem it can be fixed by adding the stop token to your config.json manually:

{
  "title": "Llama 3",
  "provider": "ollama",
  "model": "llama3",
  "completionOptions": {
    "stop": ["<|eot_id|>"]
  }
}

I will keep this issue open until we've merged pre-release to main and given some time for everyone to upgrade

[sestinj]

This is now long past merged, so closing the issue : )

I do believe that Ollama has also patched the problem, but if anyone sees it again don't hesitate to re-open