[Feature Request] Access to full trajectory and token usage for each request

[Lawhy]

Required prerequisites

• [x] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• [ ] Consider asking first in a Discussion.

Motivation

When planning budget for agentic systems, we would like to see which request along the trajectory costs more tokens than others. The token usage of each base model request can be a good signal for budget planning, reward model, and more. Would like to see CAMEL support easy access to full trajectory and token usage for each request along the trajectory, attributing to individual agents and tool calls.

@boerz-coding please add some specific examples to describe the gap.

Solution

No response

Alternatives

No response

Additional context

No response

[boerz-coding]

I have submitted two sub-issues to further clarify our request in this current issue.

[Wendong-Fan]

thanks @boerz-coding !

[lllllyh01]

Hi @Wendong-Fan , I can take a closer look into this feature. Can I take this one?

[Saedbhati]

@lillyh01 I am also working on these. We can collaborate on these issue

[Wendong-Fan]

thanks @lllllyh01 and @Saedbhati ! issue assigned to you

[lllllyh01]

@Saedbhati Awesome! I'd love to collaborate and discuss further about this issue. I'm currently exploring possible solutions, and let's stay in touch for any ideas we have :)

[boerz-coding]

@lllllyh01 @Saedbhati Thank you. I am also happy to

1. provide more context / example code script that we are running where we different tools are called, if necessary.
2. discuss solutions.

[Saedbhati]

@lllllyh01 you can drop your email/discord so we can have discussion on this issue. Thanks

[lllllyh01]

@Saedbhati here is my discord: yihan_lin Thanks!

[Wendong-Fan]

hey @Saedbhati and @lllllyh01 , thanks for taking this issue! Please feel free to share any updates from your discussion here. If there's anything I can do to support you, don't hesitate to tag me

[lllllyh01]

Thanks @Wendong-Fan . We're still aligning calendars for a short sync; to avoid blocking progress, I started on sub-issue #3219 first. I'm preparing a draft commit for it and I'll try to finish it within a couple days.

@Saedbhati please feel free to share any updates from your side, or we can also schedule a quick sync meeting by your preference

[Wendong-Fan]

thanks @lllllyh01 , left some comments for the PR, feel free to check

[lllllyh01]

Hi @Wendong-Fan , thank you for your review. I've updated this PR and left some responses for your comments. Please feel free to review, and let me know any further feedbacks.

Another thing I would like to bring up is that I found another PR #3325 from @LIHUA919 also works on this issue. I think we are technically on the same track on solving this issue. Since this PR is earlier than mine, to avoid redundancy I am considering migrate my changes to #3325 .