agents icon indicating copy to clipboard operation
agents copied to clipboard

Expose metrics from MultimodalAgent and RealtimeModel

Open davidzhao opened this issue 1 year ago • 2 comments

davidzhao avatar Oct 30 '24 06:10 davidzhao

It looks like some basic considerations were made for the realtime model with a Usage model defined here: https://github.com/livekit/agents/blob/22f06bf321532a92e12b7a469692f0c8e5503dd4/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/api_proto.py#L144

I just wanted to add that the exposed usage details should include input_token_details and output_token_details. Without those details the 3 existing token numbers are not useful because cached tokens, text tokens, and audio tokens are all priced differently.

Sample usage detail

usage: {
	total_tokens: 330
	input_tokens: 181
	output_tokens: 149
	input_token_details: {
		cached_tokens: 0
		text_tokens: 181
		audio_tokens: 0
	}
	output_token_details: {
		text_tokens: 34
		audio_tokens: 115
	}
}

Thanks in advance

yuyuma avatar Nov 03 '24 00:11 yuyuma

please feel free to submit a PR :)

davidzhao avatar Nov 03 '24 05:11 davidzhao

@davidzhao done, thanks!

yuyuma avatar Nov 04 '24 22:11 yuyuma

MultimodalAgent metrics are in #1080

davidzhao avatar Nov 14 '24 05:11 davidzhao