`composeContext` function omits memories when exceeding 50 items with SQLite adapter, leading to incorrect state data

[snobbee]

Describe the bug

When more than 50-54 memory items are created within the same room ID, the composeContext function occasionally omits certain memories, leading to incomplete state data. This issue occurs when using the SQLite adapter. Note that I have not tested this with the other database adapters available, but I would be surprised if an SQLite database cannot handle more than 50 records within the memories table.

To Reproduce

Steps to reproduce the behavior:

1. Clone the repository:

   git clone [email protected]:Sifchain/sa-eliza.git
   

2. Checkout the branch realitySpiral/testUsefulWorkflows (based on the develop branch from the upstream eliza repository):

   git checkout realitySpiral/testUsefulWorkflows
   

3. Set the following .env variables:

   GITHUB_API_TOKEN=ghp_XXX
   
   GITHUB_OWNER=sifchain
   GITHUB_REPO=sa-eliza
   GITHUB_BRANCH=sif-dev
   
   GITHUB_ISSUES_LIMIT=100
   GITHUB_PULL_REQUESTS_LIMIT=100
   

   Refer to this guide for generating and using the GitHub API token.

4. Install dependencies and start the agent:

   pnpm install -r
   rm -rf agent/data && pnpm build && pnpm start --character=characters/staff-engineer.character.json
   

5. Observe the logs. At some point, you will notice the following:

   ["\u2139 Executing handler for action: INITIALIZE_REPOSITORY"]
   
   ◎ LOGS
     [initializeRepository] Composing state for message:
     {"id":"6a74eb9e-2eb1-00b7-bce3-a2d819d53bd5","userId":"e0705f05-5af9-0ac4-80b9-a55815900197","agentId":"b1ccf0ce-60b8-00ed-8da1-0fd7b791d233","content":{"text":"No memories found, starting to initialize repository and create memories.","action":"NOTHING","source":"github","inReplyTo":"3b2b2818-d716-08b7-a767-418a416a89c7"},"roomId":"a5e95f21-99e3-0354-add8-79108740b31a","createdAt":1736270012868}
   
   ["\u2139 Initializing repository realityspiral/test on branch main..."]
   
   ["\u2139 Repository path: /tmp/elizaos-repos/realityspiral/test"]
   
   ["\u2139 Repos directory already exists: /tmp/elizaos-repos/realityspiral"]
   
   ["\u2139 Cloning or pulling repository realityspiral/test... @ branch: main"]
   
   ["\u2139 URL: https://github.com/realityspiral/test.git @ branch: main"]
   

   The repository information (realityspiral/test) is incorrect.

6. Update the .env file to reduce memory storage limits:

   GITHUB_ISSUES_LIMIT=10
   GITHUB_PULL_REQUESTS_LIMIT=10
   

7. Restart the agent:

   rm -rf agent/data && pnpm build && pnpm start --character=characters/staff-engineer.character.json
   

8. Observe the logs again. This time, the correct repository information is used:

   ["\u2139 Executing handler for action: INITIALIZE_REPOSITORY"]
   
   ◎ LOGS
     [initializeRepository] Composing state for message:
     {"id":"53d09a5a-289e-03d6-a49e-0fb655fe55c2","userId":"41b1749c-25aa-0dc6-9cdf-cf3a1141fc55","agentId":"b1ccf0ce-60b8-00ed-8da1-0fd7b791d233","content":{"text":"No memories found, starting to initialize repository and create memories.","action":"NOTHING","source":"github","inReplyTo":"3b2b2818-d716-08b7-a767-418a416a89c7"},"roomId":"a5e95f21-99e3-0354-add8-79108740b31a","createdAt":1736270165808}
   
   ["\u2139 Initializing repository sifchain/sa-eliza on branch sif-dev..."]
   
   ["\u2139 Repository path: /tmp/elizaos-repos/sifchain/sa-eliza"]
   
   ["\u2139 Repos directory already exists: /tmp/elizaos-repos/sifchain"]
   
   ["\u2139 Cloning or pulling repository sifchain/sa-eliza... @ branch: sif-dev"]
   
   ["\u2139 URL: https://github.com/sifchain/sa-eliza.git @ branch: sif-dev"]
   
   ["\u2139 Checking out branch sif-dev in repository /tmp/elizaos-repos/sifchain/sa-eliza"]
   
   ["\u2139 Repository initialized successfully! URL: https://github.com/sifchain/sa-eliza @ branch: sif-dev"]
   

Expected behavior

The composeContext function should include all relevant memories without omitting any, regardless of the number of items stored in memory. Additionally, the correct repository information should always be used.

Screenshots

N/A

Additional context

• This issue appears to be tied to the SQLite adapter’s handling of large memory sets.
• Reducing the memory limits in the .env file mitigates the issue but does not resolve the underlying problem.
• This behavior could impact other functionalities relying on the composeContext function if memory limits are exceeded.