Add optional grep parameter to all tools for context window optimization

[aantn]

Motivation

Currently, when HolmesGPT runs investigations, it often loads all relevant data upfront from tools, which can lead to large context windows. In contrast, coding agents like Claude Code typically start with grep operations or read limited file subsets to keep context manageable, then requests more data as needed.

We should try the same approach for us, to see if it helps with keeping context windows from overflowing, thereby causing truncation.

General Idea

We should add an optional grep parameter to every tool that allows the LLM to filter tool outputs on-demand. This would enable the LLM to:

• Request only relevant portions of large tool outputs initially
• Follow up with additional grep patterns or full data as needed

Implementation Approach

This could potentially be implemented using the upcoming transformers API as a "grep transformer" that:

• Adds grep parameters visible to the LLM for all tools
• Applies filtering to tool outputs based on LLM-specified patterns
• Maintains backward compatibility with existing toolsets

[mrchocha]

Hey @aantn, just to confirm my understanding, the issue seems to be that we're hitting context size limitations because all tools are being loaded into the context upfront. We're exploring a way to load tools dynamically to address this, right?

I came across this approach it might be helpful https://www.youtube.com/watch?v=3ISRS2hQlfI

[mrchocha]

Hey @aantn, just to confirm my understanding, the issue seems to be that we're hitting context size limitations because all tools are being loaded into the context upfront. We're exploring a way to load tools dynamically to address this, right?

I came across this approach it might be helpful https://www.youtube.com/watch?v=3ISRS2hQlfI

If this is the way you are looking for then can i give it a try? would love to work on this.

[aantn]

@mrchocha not exactly! The issue here is a bit tricker and its about the actual output from tools exceeding context windows. Its probably not the best beginner issue to look at!