Code Llama tab completion having EOT Token present

[arthurcohen]

Before submitting your bug report

• [ ] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: MacOS Sonoma 14.3.1
- Continue: v0.9.65 (pre-release)
- IDE: VSCode 1.86.2
- Model: codellama:7b-code

Description

After setting up Continue with the Ollama provider, I enabled Tab Autocomplete and it mostly works fine. The issue is that the autocomplete feature is always adding at the end an <EOT> regardless of the settings I tried using. I believe this string is called End-Of-Text token, and it is referenced on the tokenization file inside the model repository.

The issue seems to happen with suggestions that usually come up as one-liners and multi-line suggestions are showing up fine so far.

To reproduce

1. Install Ollama
2. Pull the codellama model: ollama pull codellama:7b-code
3. Install Continue on VSCode
4. Setup Ollama provider integration: 4.1 For reference

{
  "models": [
    {
      "model": "AUTODETECT",
      "title": "Ollama",
      "completionOptions": {},
      "apiBase": "http://localhost:11434",
      "provider": "ollama"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Tab Autocomplete Model",
    "provider": "ollama",
    "model": "codellama:7b-code",
  },
  ...
}

5. Go to any file and start typing, you'll see this:

const UserSchema = object({
  id: z.number(),
  name: z.string(),
  lastName: z.string(), <EOT> // this was added by the autocomplete
  email: z.string(),
  picture: z.string(),
  password: z.string(),
})

Log output

No response