feat: Add table viz and persistence to Metrics
Is your feature request related to a problem?
In the current implementation of our new UDF metrics, we currently emit metrics in the swordfish TUI inline with the UDF process. When processing completes and a Dataframe is shown, the metrics are lost from the display.
Then ...
There are a couple of ways this could be accomplished but lets break down the use case and requirements:
- Users add metrics to UDFs to gain observability into their processes.
- Having live metrics emitted it great, but only if it's also persist-able for further analysis across runs.
- Additionally, it's critical that Users are able to see the final metrics for each run at the end.
Describe the solution you'd like
Ideally, we wouldn't lose the metrics and we could somehow display them once the run completed.
The most straightforward way to do this would be by simply printing the metrics in the same manner it is displayed in the TUI as plain text:
╭────────────────────────────────────────┬───────────────────────╮
│ quote ┆ nemotron-response │
╞════════════════════════════════════════╪═══════════════════════╡
│ I'm going to be the next Hokage! ┆ **Show:** *Naruto* │
│ ┆ **Character:** N… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ I am going to be the king of the pira… ┆ **Show:** One Piece │
│ ┆ **Character:** … │
╰────────────────────────────────────────┴───────────────────────╯
(Showing first 2 of 2 rows)
16 rows in, 9 rows out, 413 tokens_in, 1,374 tokens_out, 1787 total_tokens, 9 num_requests
Alternatively it would be more more visually appealing if we could just visualize metrics as another table following the results like:
> Results
╭────────────────────────────────────────┬───────────────────────╮
│ quote ┆ nemotron-response │
╞════════════════════════════════════════╪═══════════════════════╡
│ I'm going to be the next Hokage! ┆ **Show:** *Naruto* │
│ ┆ **Character:** N… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ I am going to be the king of the pira… ┆ **Show:** One Piece │
│ ┆ **Character:** … │
╰────────────────────────────────────────┴───────────────────────╯
(Showing first 2 of 2 rows)
> Metrics
╭─────────┬──────────┬───────────┬────────────┬──────────────┬──────────────╮
│ rows_in ┆ rows_out ┆ tokens_in ┆ tokens_out ┆ total_tokens ┆ num_requests │
╞═════════╪══════════╪═══════════╪════════════╪══════════════╪══════════════╡
│ 16 ┆ 9 ┆ 413 ┆ 1374 ┆ 1787 ┆ 9 │
╰─────────┴──────────┴───────────┴────────────┴──────────────┴──────────────╯
Another approach would be to simply label metrics as a map since all metrics are cast to map[str, float]. @srilman visualizing it like you might in excel.
Describe alternatives you've considered
No response
Additional Context
https://github.com/Eventual-Inc/Daft/pull/5549
Would you like to implement a fix?
No