microsoft
Will trajectory modifications that bypass the LLM API be lost from the trajectory and thus unavailable for RL training?
In my RAG agent’s “think → search → think…” chain of operations, I sometimes edit the tokens generated by the LLM (e.g., deleting a few tokens, reordering them, etc.) before constructing the next action.
These edits never touch the LLM API and therefore do not create new spans.
Does this mean that the final trajectory still retains the original (pre-edit) token IDs? Or have I missed some hook that would allow me to overwrite the captured prompt/response_token_ids with the edited tokens?
If the current tracer cannot detect such modifications, what is the officially recommended solution—is it to trigger a dummy LLM call, register an MCP tool, or something else?
Thank you!
Alternatively, is there a simpler approach where I can treat the entire process as a black box to obtain the final trajectory, then use some post-processing function to parse and convert the trajectory into a format understandable by the training server for RL training—requiring me only to provide reward values on the agent client side?
If the current tracer cannot detect such modifications, what is the officially recommended solution—is it to trigger a dummy LLM call, register an MCP tool, or something else?
You can use an agentops decorator: https://docs.agentops.ai/v1/concepts/decorators and then use TripletExporter to control the triplet generation logic.
Alternatively, is there a simpler approach where I can treat the entire process as a black box to obtain the final trajectory, then use some post-processing function to parse and convert the trajectory into a format understandable by the training server for RL training—requiring me only to provide reward values on the agent client side?
Yes. You can directly return the trace and triplets as the return value of training_rollout: https://github.com/microsoft/agent-lightning/blob/5724f63cfc75bcc2f4fb56958ef384d307717c18/agentlightning/runner.py#L127
