All-Hands-AI
[Bug]: LLM Completion Tokens for LLMSummaryCondensator are not included in trajectory.json
Is there an existing issue for the same bug? (If one exists, thumbs up or comment on the issue instead).
- [x] I have checked the existing issues.
Describe the bug and reproduction steps
It appears that there are two condensation llm-calls happening and neither are being tracked in the trajectory.json.
I discovered this when debugging the LLMSummarizingCondenser.
I start a new conversation on my test-project: https://github.com/happyherp/openhands-test
I set a breakpoint in CodeActAgent._step and wait for it to be hit. When hit, I set self.condenser.max_size to 10 to more quickly reach a condensation. Then I disable the breakpoint.
I set a breakpoint in LLMSummarizingCondenser: get_condensation
I ask the agent to read all files. This triggers the breakpoint in LLMSummarizingCondenser: get_condensation.
I note the value of self.llm.metrics.get()
{
"model": "anthropic/claude-3-7-sonnet-20250219",
"prompt_tokens": 1307,
"completion_tokens": 140,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"response_id": "chatcmpl-040366cb-cf32-4732-8e8b-1160eb01bed2"
}
I also note that summary_event is AgentCondensationObservation('No events summarized') I note the value of summary I let the program resume.
Immediately the breakpoint is hit again.
now the summary-event contains the event that was created from the previous call.
self.llm.metrics.get() now contains both the previous and the current llm-call.
"token_usages": [
{
"model": "anthropic/claude-3-7-sonnet-20250219",
"prompt_tokens": 1307,
"completion_tokens": 140,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"response_id": "chatcmpl-040366cb-cf32-4732-8e8b-1160eb01bed2"
},
{
"model": "anthropic/claude-3-7-sonnet-20250219",
"prompt_tokens": 6710,
"completion_tokens": 226,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"response_id": "chatcmpl-a4978e4a-0220-45f6-b0f8-15aac015884b"
}
]
I let the program resume. because of the low value, of condenser.max_size, another condensation happens later on. I stop the agent and download the trajectory.json and compare it to what I saw when debugging.
Before the condensation we have events 14(RunAction) and 15(RunObservation).
Event 14 has
"llm_metrics": {
"accumulated_cost": 0.03647205,
"accumulated_token_usage": {
"model": "anthropic/claude-3-7-sonnet-20250219",
"prompt_tokens": 17488,
"completion_tokens": 410,
"cache_read_tokens": 17466,
"cache_write_tokens": 6671,
"response_id": ""
}
Then comes event 16 CondensationAction
{
"id": 16,
"timestamp": "2025-04-16T10:05:31.233715",
"source": "agent",
"message": "Summary: <SUMMARY>\nUSER_CONTEXT: ....", //truncated
"action": "condensation",
"llm_metrics": {
"accumulated_cost": 0.03647205,
"accumulated_token_usage": {
"model": "anthropic/claude-3-7-sonnet-20250219",
"prompt_tokens": 17488,
"completion_tokens": 410,
"cache_read_tokens": 17466,
"cache_write_tokens": 6671,
"response_id": ""
},
"costs": [],
"response_latencies": [],
"token_usages": []
},
"args": {
"forgotten_event_ids": null,
"forgotten_events_start_id": 5,
"forgotten_events_end_id": 12,
"summary": "<SUMMARY>\nUSER_CONTEXT: ....", //truncated
"summary_offset": 1
}
}
Note that the field token_usages is empty, and accumulated_token_usage is exactly the same as in the event 14 before the condensation.
The prompt_tokens values(1307, 6710) that I saw when debugging in self.llm.metrics.get() do not appear anywhere in the trajectory-file. Neither does their sum 8017.
OpenHands Installation
Development workflow
OpenHands Version
e69ae81ad2392b4e11be4ed44a9965aa192f2487 main
Operating System
WSL on Windows
Logs, Errors, Screenshots, and Additional Context
The trajectory: trajectory-26b0c7aa6cba4c06814ed112b501ac43.json
The core of the issue seems to be how event.metrics is set.
In AgentController
def _prepare_metrics_for_frontend(self, action: Action) -> None:
"""Create a minimal metrics object for frontend display and log it.
To avoid performance issues with long conversations, we only keep:
- accumulated_cost: The current total cost
- accumulated_token_usage: Accumulated token statistics across all API calls
Args:
action: The action to attach metrics to
"""
# Create a minimal metrics object with just what the frontend needs
metrics = Metrics(model_name=self.agent.llm.metrics.model_name)
metrics.accumulated_cost = self.agent.llm.metrics.accumulated_cost
metrics._accumulated_token_usage = (
self.agent.llm.metrics.accumulated_token_usage
)
action.llm_metrics = metrics
I debugged this code and saw that the agent.llm and agent.condenser.llm are two different objects, each keeping track metrics on their own. I think that should stay that way. We should not assume that just one llm is being used.
So to fix this, I will try to see if I can sum up the accumulated metrics of both agent.llm and condenser.lllm.
