[CON-279] System message from prompt file present twice in prompt logs

[sestinj]

trafficstars

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:*
- Continue:0.9.215
- IDE:*

Description

From https://discord.com/channels/1108621136150929458/1291763559876919406:

Following https://docs.continue.dev/customize/deep-dives/prompt-files I noticed that the system message defined in my test.prompt file is present twice in the final formatted prompt sent to the chat LLM.

For instance, "You are an expert programmer" is twice in the LLM prompt logs in VSCode. I have tested with a local ollama and a remote azure chat model. Same behaviour. Also in pycharm: there is no prompt logs (as far as I know), but looking at the logs of my local ollama instance (started with the debug flag), the system message also appears twice.

To reproduce

No response

Log output

No response

_CON-279

[sestinj]

Related to https://github.com/continuedev/continue/blob/dev/core/config/promptFile.ts#L103-L119 if I had to guess

[StructByLightning]

I was able to track down the issue to line 358 of countTokens.ts in compileChatMessages(). This function concatenates two system messages together. This update fixes it:

function compileChatMessages(
  modelName: string,
  msgs: ChatMessage[] | undefined,
  contextLength: number,
  maxTokens: number,
  supportsImages: boolean,
  prompt: string | undefined = undefined,
  functions: any[] | undefined = undefined,
  systemMessage: string | undefined = undefined,
): ChatMessage[] {
  const msgsCopy = msgs
    ? msgs
        .map((msg) => ({ ...msg }))
        .filter((msg) => msg.content !== "" && msg.role !== "system")
    : [];

  if (prompt) {
    const promptMsg: ChatMessage = {
      role: "user",
      content: prompt,
    };
    msgsCopy.push(promptMsg);
  }

  if (
    (systemMessage && systemMessage.trim() !== "") ||
    msgs?.[0].role === "system"
  ) {
    let content = "";
    if (msgs?.[0].role === "system") {
      content = stripImages(msgs?.[0].content);
    }
    if (!content && systemMessage && systemMessage.trim() !== "") { //add check to only concatenate systemMessage if one didn't already exist
      const shouldAddNewLines = content !== "";
      if (shouldAddNewLines) {
        content += "\n\n";
      }
      content += systemMessage;
    }
    const systemChatMsg: ChatMessage = {
      role: "system",
      content,
    };
    // Insert as second to last
    // Later moved to top, but want second-priority to last user message
    msgsCopy.splice(-1, 0, systemChatMsg);
  }

I can open a PR for this but this is my first time doing anything with the Continue codebase and I'm unsure if this will have unwanted side effects elsewhere (presumably it was designed the way it was for a reason).

[github-actions[bot]]

This issue hasn't been updated in 90 days and will be closed after an additional 10 days without activity. If it's still important, please leave a comment and share any new information that would help us address the issue.

[github-actions[bot]]

This issue was closed because it wasn't updated for 10 days after being marked stale. If it's still important, please reopen + comment and we'll gladly take another look!