[BUG] Trained data not being used

[mrSingh007]

Description

I have performed some train iteration and found that still when running 'crewai run' trained data not being considered. Doesn't matter which LLM model is being used the result is not being changed according to trained data.

Used commands: To train crewai train -n 2 -f lorem-train.pkl To run `crewai run``

Steps to Reproduce

How to reproduce ?

1. in new very new project. (using any llm provider. I tested with llama and gemini)
2. in reporting_task under add "A well-organized written report." as expected_output

Now start at least 2 iterations of training and for reporting task give feedback to not include 'conclusion' section and to use numbering for all sections.

After training start now normal run with 'crewai run' on same default topic. You will find that now report is being generated as trained during training. (I just provided simple example here, you could play around on feedback).

After trying many attempts found that result is not being changed.

Expected behavior

According to provided reproduce steps: The generated report should contain numbering and should not include 'conclusion' section

Screenshots/Code snippets

NA

Operating System

Other (specify in additional context)

Python Version

3.12

crewAI Version

0.117.1

crewAI Tools Version

0.117.1,<1.0.0

Virtual Environment

Venv

Evidence

NA

Possible Solution

NA

Additional context

[mouramax]

Correct me if I'm wrong, but I think your issue might be the same one covered in this forum thread?

If that's the case, it's the same problem: your training file needs to be named exactly trained_agents_data.pkl, because that value is hardcoded in crewai/utilities/constants.py.

[mrSingh007]

@mouramax You're right — the file is expected to be named trained_agents_data.pkl, but it still behaves inconsistently. Sometimes it works, and sometimes it doesn't.

I trained the data and renamed the file, and it worked — it produced the expected output using the trained data. Then I deleted the .pkl file, retrained the LLM, and renamed the new file, but now it's not recognizing the trained data.

When I print data.get(self.role), it has data the first time, but in the second test, it returns None.

if data := CrewTrainingHandler(TRAINED_AGENTS_DATA_FILE).load():
    print(data.get(self.role))
    if trained_data_output := data.get(self.role):

Sometimes it generates two .pkl files, and sometimes only one. It's really flaky! @joaomdmoura

[mikhail]

Correct me if I'm wrong, but I think your issue might be the same one covered in this forum thread?

If that's the case, it's the same problem: your training file needs to be named exactly trained_agents_data.pkl, because that value is hardcoded in crewai/utilities/constants.py.

Could this be added to documentation? Currently there's no explanation of how training actually works and impacts the run.

[lucasgomide]

@mikhail 100% Could you submit a PR with that? I'd appreciate that

[mikhail]

@lucasgomide I'd be happy to if I knew how it works! I'm not actually sure what it does. Is there some RAG system or does it append to the prompt?

[mrSingh007]

why this has been closed ? is it fixed ? @lucasgomide

[lucasgomide]

why this has been closed ? is it fixed ? @lucasgomide

To my understanding your issue was resolved here - when moura says: use the proper training_file. Let me know if I'm wrong, please.

[lucasgomide]

@lucasgomide I'd be happy to if I knew how it works! I'm not actually sure what it does. Is there some RAG system or does it append to the prompt?

Actually, I meant to enhance our docs, since you have pointed it "Could this be added to documentation". So it would only involve updating some docs file (the .mdx file)

Do you have any other insights to push? Maybe I didn't get your thoughts

[mrSingh007]

@lucasgomide as i mentioned above its very flaky. sometime its works and other time now. With Ollama it does not even work at all. That the error coming when training ollama model

 Training failed: Failed to convert text into a Pydantic model due to error: 3 validation errors for TrainingTaskEvaluation
suggestions
  Field required [type=missing, input_value={'properties': {'suggesti...tain its performance."}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/missing
quality
  Field required [type=missing, input_value={'properties': {'suggesti...tain its performance."}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/missing
final_summary
  Field required [type=missing, input_value={'properties': {'suggesti...tain its performance."}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/missing
....
```

[lucasgomide]

I gotcha.. so it turns out to be a different issue than we initially thought.. it originally started as "Trained data not being used". No worries open it again, tho. Tks for pushing but

[mrSingh007]

@lucasgomide
Seems like there is something wrong..

• if not using ollama models then its not using the trained data. it do create the pkl files but only works 1% of time. Mean if i performed 20 times the training (deleting pkl files and did retraining from zero) then maybe only once it would work that it using pkl files.
• With ollama it doesn not work at all and throw above mentioned error.

[lucasgomide]

thanks for the detailed explanation I'll try to look into it over the next few days 🙏🏼

[mouramax]

@lucasgomide, while you're digging into this, it's really key to remember that "Ollama models" are almost always these smaller, simpler types that seriously struggle with following instructions. The error the OP mentioned is all about the generation of structured output at some point during the training process.

Just today, in this thread, a user was having a heck of a time getting structured output (be it Task.output_json or Task.output_pydantic) just because of one extra instruction in Task.expected_output – and get this, she was even using a 70B model!

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[mrSingh007]

@lucasgomide what is status here ?

[lucasgomide]

@lucasgomide what is status here ?

I planning to back to it this week

[lucasgomide]

@mrSingh007 today we merged today this PR that improves the data training system for those small/medium moldes.