Malformed JSON Error with codegemma-7b and Hugging Face TGI in VS Code

[CMobley7]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Mac 14.7
- Continue: v0.8.52
- IDE: VS Code 1.93.1
- Model: `codegemma-7b` (served via Hugging Face TGI on port 80)
- config.json:
  
{
  "models": [
    {
      "title": "CodeGemma Chat",
      "provider": "huggingface-tgi",
      "model": "codegemma-7b-it",
      "apiBase": "http://ip_address
    }
  ],
  "tabAutocompleteModel": {
    "title": "CodeGemma Code Completion",
    "provider": "huggingface-tgi",
    "model": "codegemma-7b",
    "apiBase": "http://ip_address"
  },
  "tabAutocompleteOptions": {
    "maxPromptTokens": 1000,
    "useCache": true,
    "multilineCompletions": "auto",
    "debounceDelay": 200
  },
  "allowAnonymousTelemetry": false,
  "customCommands": [
    {
      "name": "test",
      "prompt": "{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
      "description": "Write unit tests for highlighted code"
    }
  ],
  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "edit",
      "description": "Edit selected code"
    },
    {
      "name": "comment",
      "description": "Write comments for the selected code"
    },
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ]
}

Description

I am encountering a persistent Malformed JSON sent from server: {"error":"Input validation error: `stop` supports up to 4 stop sequences. Given: 16","error_type":"validation"} error when using the codegemma-7b model for tab autocompletion with the Hugging Face TGI provider in VS Code. The error message indicates an issue with the stop parameter exceeding the allowed number of stop sequences, despite various attempts to configure it correctly.

Additional context

• Both my models (codegemma-7b for autocompletion and codegemma-7b-it for chat) are running through Hugging Face TGI on port 80.
• The chat feature with codegemma-7b-it works without any issues.
• The problem seems isolated to the tab autocompletion functionality with codegemma-7b.

Possible issue

While it's most likely user error, it appears that Continue might not be correctly handling the stop sequences defined in the codegemmaFimTemplate when the codegemma or gemma template is used. This could lead to Continue sending an incorrect number of stop sequences in the request to the TGI server, causing the "Malformed JSON" error.

To reproduce

1. Initial Configuration: Started with the following basic configuration in config.json:

"tabAutocompleteModel": {
    "title": "CodeGemma Code Completion",
    "provider": "huggingface-tgi",
    "model": "codegemma-7b",
    "apiBase": "http://ip_address" 
  }

2. Gradual Adjustments: Made the following incremental changes to the configuration based on troubleshooting suggestions, documentation, and analysis of the codegemmaFimTemplate:

   • Added "template": "codegemma" (also tried "template": "gemma")
   • Set "maxStopWords": 4 (later changed to 2 and then 1)
   • Modified "completionOptions.stop" to various combinations, including:
     • ["<|fim_prefix|>", "<|fim_suffix|>", "<|fim_middle|>", "<|file_separator|>", "<end_of_turn>", "<eos>"]
     • ["<end_of_turn>", "<eos>"]
     • ["<eos>"]

3. Error persists: The same "Malformed JSON" error occurred after each configuration change, with the number of given stop sequences varying (e.g., 15 in the initial error message).

Expected behavior

The tab autocompletion should function correctly without any "Malformed JSON" errors.