bug: All LLMs except Claude ignores the command output

[mattia-consiglio]

Version

1.89.1 (user setup)

Describe the bug

When running the custom command

"commit-message": {
	"description": "Commit message for staged changes",
	"prompt": "Suggest an informative commit message by summarizing code changes from the shared command output. The commit message should follow the conventional commit format and provide meaningful context for future readers. The commit summary that includes the commit type (scope) has to be under 50 characters including spaces. Be sure to include the commit summary. Use bullet points for the description. The commit message should be presented in a code block.",
	"context": {
		"selection": false,
		"openTabs": false,
		"currentFile": false,
		"command": "git diff --staged --minimal -w"
	}
}

There are 3 different behavior:

1. GPT-3.5 Turbo, Mixtral (all), Ollama: The LLM recognize that has no access to the output of the terminal.
2. GPTP-4 Turbo: The LLM hallucinate the response giving a fake commit message, without saying it.
3. Claude (all): The LLM gives an appropriate answer.

Expected behavior

The LLM to be able to use the command output.

Additional context

No response

[jdorfman]

Thanks, @mattia-consiglio. I will get some eyes on this.

[github-actions[bot]]

This issue is marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed automatically in 5 days.