Chat history not being properly recorded

[myanvoos]

I ran a few experiments in agentic mode today (with the --ag flag) and found that the samples' log.txt are not showing the agents' interactions that were logged to the console. At the end of the experiments the log.txt for every single sample just looks like:

Prototyper

SemanticAnalyzer
{'has_err': False, 'err_type': 'NO_SEMANTIC_ERR', 'crash_symptom': '', 'crash_stacks': [], 'crash_func': {}}

Experiments without agentic mode similarly seem to have a blank chat history. They only show the initial prompt in log.txt.

After some digging into the codebase and adding

def _container_tool_reaction(self, cur_round: int, response: str,
                               build_result: BuildResult) -> Optional[Prompt]:
    """Validates LLM conclusion or executes its command."""
    prompt = prompt_builder.DefaultTemplateBuilder(self.llm, None).build([])

    if response:
      # Adding this
      build_result.chat_history[self.name] += f"\nRound {cur_round}:\n{response}"

to prototyper.py, this now works as expected and all the log.txt are showing the Prototyper's responses in the newest batch of agentic experiments. However I'm not sure if this is the most idiomatic way to go about it.

It would be great if you can check the log.txt for cloud experiments @DonggeLiu, just to see if you're seeing this same behaviour as well.

[DonggeLiu]

Thanks for reporting this, @myanvoos !

This is a good catch, most of our experiments are run on cloud so we do need more help maintain local exps. Chat history works fine in cloud exps, it is set here: https://github.com/google/oss-fuzz-gen/blob/358aff99ac5a4560acc537b726b0c581eb1df1bd/common/cloud_builder.py#L417

Experiments without agentic mode similarly seem to have a blank chat history. They only show the initial prompt in log.txt.

Prototyper is an agent, we indeed missed its chat history, your fix should work. SemanticAnalyzer is not really an agent (LLM does not interact with tools), hence its log is set manually. We will likely replace it with a crash analyzer.

[myanvoos]

Thanks for reporting this, @myanvoos !

This is a good catch, most of our experiments are run on cloud so we do need more help maintain local exps. Chat history works fine in cloud exps, it is set here:

oss-fuzz-gen/common/cloud_builder.py

Line 417 in 358aff9 result.chat_history = {agent.name: cloud_build_log}

Experiments without agentic mode similarly seem to have a blank chat history. They only show the initial prompt in log.txt.

Prototyper is an agent, we indeed missed its chat history, your fix should work. SemanticAnalyzer is not really an agent (LLM does not interact with tools), hence its log is set manually. We will likely replace it with a crash analyzer.

Yep, I figured that it's probably an issue with local experiments since I did see the chat history clearly in the trial/sample page screenshot you shared. Also I tried again with a cloud experiment since opening the issue and the chat logs did show up correctly there.

All good, I'll go in and make the fix later on the Prototyper (and possibly other agents as well).

Thanks for letting me know!

[DonggeLiu]

Thank you again for reporting this to us and spending time&effort verifying this locally!