JackHopkins
Timing metrics and tracker
Add a tracker for timing metrics. This is essential for timing-sensitive experiments and eventually for getting real-time performance. Some relevant changes:
- claude and async openai clients (used for more than just openai models) are now set with max_retries=0 so that all retry logic is kept within
llm_factory.pyand hence trackable. - timing metrics are printed in stdout for now (see screenshot below). Retry timing is not explicitly captured but accumulated as 'unaccounted' time and shown as such in the totals.
Programhas a new timing_metrics field. Needs to be updated in DB.
In this example, there was one retry (seen as a 529 at the top). The timing metrics show two distinct calls to claude. There is 2s of unaccounted time which came from the tenacity retry logic waiting for 2 seconds.
Looks good. We have a field of elapsed time also being returned from the actions in the client.py which may be useful.
Can you clarify where is this elapsed time being returned? Having trouble finding it.
It was in controller.py but it seems to have been removed from the object we send back. Should be straightforward to turn it back on.
The variable is ‘elapsed’. Which is the execution time of the tool in milliseconds (round trip).
On 27 May 2025, at 11:06, Neel Kant @.***> wrote:
kantneel left a comment (JackHopkins/factorio-learning-environment#207) https://github.com/JackHopkins/factorio-learning-environment/pull/207#issuecomment-2911945631 Can you clarify where is this elapsed time being returned? Having trouble finding it.
— Reply to this email directly, view it on GitHub https://github.com/JackHopkins/factorio-learning-environment/pull/207#issuecomment-2911945631, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA5T6OXJLCCXRAWWFMJ7USD3AQ2L3AVCNFSM6AAAAAB53MFKESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMJRHE2DKNRTGE. You are receiving this because you commented.