[agent] Long-term memory with repo.md

[xingyaoww]

• [ ] This change is worth documenting at https://docs.all-hands.dev/
• [ ] Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

End-user friendly description of the problem this fixes or functionality that this introduces.

---

Give a summary of what the PR does, explaining any non-trivial design decisions.

The agent is aware of the memory system:

It asks for approval BEFORE make changes to the memory file:

---

Link of any specific issues this addresses.

https://github.com/All-Hands-AI/OpenHands/issues/7302

---

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:5af04aa-nikolaik   --name openhands-app-5af04aa   docker.all-hands.dev/all-hands-ai/openhands:5af04aa

[enyst]

Thank you! Worth doing IMHO.

FWIW I was playing with something like this: repo.md gist

[xingyaoww]

@enyst appreciate you sharing the prompt! looks great - I can adapt my current version with yours! (current version is written by sonnet 3.7 lol)

[turian]

@xingyaoww it's worth providing guidance how:

1. If you don't want it every to modify repo.md that is allowed and the internal instructions allow that override.
2. There can be two levels of repo.md, one is user generated ONLY and one is machine edited too.

I find that claude code sometimes deletes important guidance from CLAUDE.md

[xingyaoww]

Sharing some progress here:

• Got a set of prompts where the agent can actually remember useful stuff (e.g., how to run test, the repo structure, etc)
• But it is trying to remember what it did in EVERY issue, and this can be a bit excessive
• Gonna play with a prompt on this a bit

Maybe a good idea would be: instead of putting the lesson learned in EACH issue in repo.md, we can adopt the idea @turian mentioned and create a two-level memory system:

• .openhands/microagents/repo.md: show agent on EVERY problem, serve as a high-level overview of things - and it can link to specific document in .openhands/memory for details (the agent can read more if it needs to)
• .openhands/memory: a folder where agents have the freedom to write notes about past experiences (e.g., specific issue it ran into earlier and how to resolve it)

[AutoLTX]

Just curious, will it be better if there is a seperate repo.md in subfolders, and 1 repo.md in a centralized folder. Read each node of repo.md from file location path in a "tree" structure. This can avoid unnecessary context and save cost. (Or just use special pattern to mark the affected folders in the repo.md)🤔I had this thought when comparing my experience with code Agent. But I haven't validated yet.

[xingyaoww]

Just curious, will it be better if there is a seperate repo.md in subfolders, and 1 repo.md in a centralized folder. Read each node of repo.md from file location path in a "tree" structure. This can avoid unnecessary context and save cost

Hey @AutoLTX - this is a good question that I don't have answer to. Since this is just the beginning for this feature, i'd intend to keep everything as simple as possible here 🤔

[xingyaoww]

@OpenHands can you help me fix failing unit test in this CI

[openhands-ai[bot]]

I'm on it! @xingyaoww can track my progress at all-hands.dev

[openhands-ai[bot]]

I've fixed the failing unit test in PR #7334 by addressing an issue with Jinja2 template loading for the repository memory feature. Here's a summary of the changes:

Problem Identified

The PR added a new repository memory feature to the codeact_agent, which included:

1. A new enable_repository_memory flag in the AgentConfig class
2. A conditional include of memory_instructions.j2 in the system prompt template
3. The unit tests were failing with a "no loader for this environment specified" error when trying to render the template

Solution Implemented

I made two key changes to fix the issue:

1. Updated the system prompt template to handle the case when agent_config is undefined:

{% if agent_config is defined and agent_config and agent_config.enable_repository_memory %}
{% include "memory_instructions.j2" %}
{% endif %}

2. Modified the PromptManager class to properly handle Jinja2 template includes:

# Create a Jinja2 environment with a FileSystemLoader
if self.prompt_dir is not None:
    from jinja2 import Environment, FileSystemLoader
    self.env = Environment(loader=FileSystemLoader(self.prompt_dir))
    self.system_template = self.env.get_template('system_prompt.j2')
    self.user_template = self.env.get_template('user_prompt.j2')
    self.additional_info_template = self.env.get_template('additional_info.j2')
    self.microagent_info_template = self.env.get_template('microagent_info.j2')

Verification

I verified the fix by running the previously failing tests:

• test_codeact_agent.py now passes all 15 tests
• test_prompt_manager.py passes all 7 tests

The changes ensure that the template engine can properly handle includes and conditional rendering based on the agent configuration.

I committed these changes to the xw/repo-memory branch, but encountered authentication issues when trying to push to the remote repository.

[xingyaoww]

supersede by https://github.com/All-Hands-AI/OpenHands/pull/8122