Crew Agents not using system message properly

[TheBitmonkey]

Is there a reason why the Crew Agents do not use the system message properly to set up their roles. I think we maybe able to improve the performance of especially local models by putting the background, role into the system prompt instead of the user role.

At the moment the user message is a mess and the models are confusing the input instead of understanding the role play and the current task. Also the system prompt should probably have the structure of response as an example of the format we want to output.

I could be wrong but I doubt I am.

[joaomdmoura]

I think the main problem with that is local models that don't have a straightforward way to set systems message as GPT-4 would, so having one solution to fit all here could be a problem, specially as they vary on their training prompt, I'm curious to hear ideas on how to work around this tho

[TheBitmonkey]

I did a workaround on my end which seems to work, although I just copied your code into my codebase, Nnt a great solution but it helps me learn whats going on under the hood. Results seem better.

I edited the prompts.py file into the following code. I am not sure if this is the correct solution or not.

from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
)

class Prompts(BaseModel):
    _prompts: Optional[Dict[str, str]] = PrivateAttr()
    language: Optional[str] = Field(
        default="en",
        description="Language of crewai prompts.",
    )

    @model_validator(mode="after")
    def load_prompts(self) -> "Prompts":
        """Load prompts from file."""
        dir_path = os.path.dirname(os.path.realpath(__file__))
        prompts_path = os.path.join(dir_path, f"prompts/{self.language}.json")

        with open(prompts_path, "r") as f:
            self._prompts = json.load(f)["slices"]
        return self

    SCRATCHPAD_SLICE: ClassVar[str] = "\n{agent_scratchpad}"

    def task_execution_with_memory(self) -> str:
        messages = [
            SystemMessagePromptTemplate.from_template(
                self._prompts["role_playing"]
                + self._prompts["tools"]
                + self._prompts["memory"]
            ),
            HumanMessagePromptTemplate.from_template(
                self._prompts["task"]
                + self.SCRATCHPAD_SLICE
            ),
        ]
        return ChatPromptTemplate.from_messages(messages)

    def task_execution_without_tools(self) -> str:
        messages = [
            SystemMessagePromptTemplate.from_template(
                self._prompts["role_playing"]
            ),
            HumanMessagePromptTemplate.from_template(
                self._prompts["task"]
                + self.SCRATCHPAD_SLICE
            ),
        ]
        return ChatPromptTemplate.from_messages(messages)

    def task_execution(self) -> str:
        messages = [
            SystemMessagePromptTemplate.from_template(
                self._prompts["role_playing"]
                + self._prompts["tools"]
            ),
            HumanMessagePromptTemplate.from_template(
                self._prompts["task"]
                + self.SCRATCHPAD_SLICE
            ),
        ]
        return ChatPromptTemplate.from_messages(messages)

[TheBitmonkey]

From digging around in the code my feelings were that it was very OpenAI centric and their models can handle the context much better that local models that certainly need a smaller context and on that need well.. better context or more specific context.

Beyond that local llms need specific language for their system message and the pre user message, post user message, etc. I would guess that a general system (for crew ai) is needed to add that as a per model basis you can add a system pre string, post string, user message pre, and post string. So as to allow the code to manage that function. (edit) So what I mean by this is that you could send in those variables easily to the same setup for lets say ollama but use different models within the same code but also use the particular strings for that model.

I believe the current langchain system of agents need to allow you to prompt in an example structure too as a prompt (obviously that consumes some of the context), to get a better output. Personally I think you could create a new class called "experts" that have a specific structure of output... ie: you tell it the structure of output as an example.

My concept of experts is similar to the the agent without tools or memory. What I am proposing by experts is that you give it an direct example of the output you are looking for. For example and specific json structure.

Don't even mind my rambling, its a great project. I don't even know what I am talking about.

[TheBitmonkey]

If I was you I would probably separate the local llm people using crewai and the Openai peps. I feel we have an opportunity right now with the poor llms to understand how best to work with them (in their current poor space) If we can get them working now then we will be so much better in the future. I think both CrewAI and langchain agents are leaning on OpenAIs models to cover the flaws that exist right now.