genkit icon indicating copy to clipboard operation
genkit copied to clipboard
Published 3 weeks ago • verified publisher icon firebase

[Tooling] Show input/output token count in Dev UI

Open apascal07 opened this issue 1 year ago • 3 comments

Describe the bug Input/output token counts are tracked but are only visible in GCP. It would be nice to see the input/output counts directly in the trace in Dev UI to get an idea for how expensive the call is during development.

Edit: Token counts are available in the output section of the trace viewer (e.g. on the model action).

  1. For the trace view, we should consider add badging and/or other UI affordances to elevate this information into clear sight.
  2. We could also consider displaying token counts on the model and prompt playgrounds as well.

apascal07 avatar Aug 19 '24 18:08 apascal07

Typically I am able to see these values in my local traces. Can you clarify?

image

MichaelDoyle avatar Aug 20 '24 03:08 MichaelDoyle

Ah, I didn't realize. I kind of expect it to appear everywhere latency does (left sidebar, right sidebar), maybe with parent nodes summing child nodes so that you can get a feel for costs at a glance just like you can with latency. WDYT?

apascal07 avatar Aug 20 '24 15:08 apascal07

Definitely - that sounds good to me (badges for tokens, etc). Thanks for clarifying.

MichaelDoyle avatar Aug 21 '24 13:08 MichaelDoyle