jan icon indicating copy to clipboard operation
jan copied to clipboard

Published 20 hours ago • verified publisher icon janhq

feat: TPS Report generator (tokens per second)

Open hwc0x01 opened this issue 1 year ago • 4 comments

Problem Would be nice to have a history/analytics of tokens/sec performance across all runs in jan over time. Should be called TPS Report because everyone loves office space.

hwc0x01 avatar Jan 05 '24 03:01 hwc0x01

Can you elaborate?

What is the main goal here? For users to see how fast certain models are? Or to find out about something else?

We can incorporate more metrics in our latest System Monitor designs: image

freelerobot avatar Jan 05 '24 07:01 freelerobot

Goals:

  1. Compare models vs other models on same prompt/assistant instruction
  2. Compare prompts vs other prompts
  3. Compare assistant instructions
  4. Jan internal: track any performance improvement/degradation with new versions of Jan/nitro

hwc0x01 avatar Jan 05 '24 18:01 hwc0x01

@0xSage I presume this would be dependent on the implementation of the latest system monitoring design?

0xgokuz avatar Jan 10 '24 12:01 0xgokuz

It would be a great pun if you actually called it a "TPS Report" 🙏

hwc0x01 avatar Jan 13 '24 17:01 hwc0x01

Moving this epic to Notion til it is confirmed: https://www.notion.so/jan-ai/System-monitor-Observability-5005bb241cdb4f94a524d20ca049b93a?pvs=4

imtuyethan avatar Mar 25 '24 09:03 imtuyethan