Llama3 chat prompt is clearly wrong

[ben-alkov]

Validations

• [X] I believe this is a way to improve. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that requests the same enhancement

Problem

Meta's definition of the correct chat template for Llama 3 is in https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3, section "Meta Llama 3 chat", which is not like anything in 'core/llm/templates/chat.ts' (except for a passing resemblance to chatmlTemplateMessages).

Worse still, the content of llama2TemplateMessages looks nothing like the recommended prompt...

(Also, AFAIU, the Instruct model's prompt should be slightly different, but I can no longer find a reference.)

Solution

Implement llama3TemplateMessages in 'chat.ts' based on https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3

[ben-alkov]

I tried the following in 'config.ts'; it seems to fix the model

• talking to itself
• giving multiple answers to a single question, but not in a logical, "alternatively, you could try..." kind of way
• ending some paragraphs with [/INST]
• sometimes endlessly repeating [/INST] when it otherwise seems to have finished answering

WARNING!!

My ECMAScript is extremely rusty, and I know absolutely nothing about TypeScript...

function llama3TemplateMessages(msgs: ChatMessage[]): string {
      if (msgs.length === 0) {
          return "";
      }
      let systemMessage = (msg) => `<|begin_of_text|><|start_header_id|>${msg.role}<|end_header_id|>\n${msg.content}<|eot_id|>\n`;
      let userPrompt = "<|start_header_id|>user<|end_header_id|>\n";
      let assistantPrompt = "<|start_header_id|>assistant<|end_header_id|>\n";
      let separator = "<|eot_id|>\n";
      let prompt = "";

      // Skip assistant messages at the beginning
      while (msgs.length > 0 && msgs[0].role === "assistant") {
        msgs.shift();
      }

      if (msgs.length > 0 && msgs[0].role === "system") {
        prompt += systemMessage(msgs.shift()!);
      }

      for (let i = 0; i < msgs.length; i++) {
        const msg = msgs[i];
        prompt += msg.role === "user" ? userPrompt : assistantPrompt;
        prompt += msg.content;
        if (i < msgs.length - 1) {
          prompt += separator;
        }
      }

      if (msgs.length > 0 && msgs[msgs.length - 1].role === "user") {
        prompt += separator;
        prompt += assistantPrompt;
      }

      return prompt;
};

export function modifyConfig(config: Config): Config {
    const model = config.models.find(
      (model) => model.title.toLowerCase().includes("llama3"),
    );
    if (model) {
      model.templateMessages = llama3TemplateMessages;
    }
  return config;
}

[ben-alkov]

Just so we're crystal clear on this: consider the above code a donation to the public domain OR as falling under Continue's license if public domain is problematic.

It's not exactly profound coding magic, but I don't want to unnecessarily create licensing issues...

[sestinj]

@ben-alkov thank you for taking the time to write this out! I've just incorporated it in this commit, and I'll release later today

[ben-alkov]

@ben-alkov thank you for taking the time to write this out! I've just incorporated it in this commit, and I'll release later today

You are very welcome - I'm just glad that it made enough sense to be useful 😆

[ryancurrah]

I tried the fix by editing ~/.continue/config.ts and it was a little better but still had <|im_end|> <|im_start|>assistant in it and returned my highlighted code back to me along with some test cases I never asked for. Not sure if I applied the fix correctly though.

Another chat I got this at the end <|im_end|> <|im_start|>user ok thanks<|im_end|>.

[ben-alkov]

@ryancurrah; Probably better to wait for the next Preview release, where it will be incorporated into Continue itself. Don't forget to reset your 'config.ts'!

[ryancurrah]

Ack I will wait. Thanks kindly.

[ben-alkov]

AFAICT, fixed in https://github.com/continuedev/continue/releases/tag/v0.9.125-vscode

[Wolfsauge]

I tried the following in 'config.ts'; it seems to fix the model [..] (model) => model.title.toLowerCase().includes("llama3"), [..]

Thanks for posting this issue and the solution. The new Llama 3 8B Instruct model now works fine for me, for example to create docstrings for Python functions using Continue in the code OSS app.

However, I also downloaded all the changes Meta made to these model config files since release.

@ryancurrah: The errors you are perceiving are due to some of the updates Meta published for their models on Huggingface not being downloaded from Huggingface to your back end yet, and/or your back end has not being properly updated to work with the Llama 3 models. Without downloading these necessary updates to the model files and or updating the back end accordingly, you will continue to see these. In other words, those errors are not related to Continue or this prompt template.