Pipeline statistics

[timonv]

Is your feature request related to a problem? Please describe. I would like to know what my pipeline costs and have data to effectively improve it.

Describe the solution you'd like After a pipeline completes, I should have access to numbers of nodes, errors, usage per model/llm, perhaps more. Would be cool if part of this data is also available via tracing.

Some musings I had on this:

• Separate aggregation thread or atomics?
• Would be nice if we can get the data while its running (apps can then display progress)
• There might be a way to estimate the total amount of nodes at any stage (depending on the type of stream)
• Also nice if the same data can be used to enhance tracing, preferably matching the beta otel llm spec

[joshka]

• metrics and autometrics crates are a couple of options to consider
• but you might want something more custom so that you can do progress and not just stats
• I have a PR a while back for tracing which extracts out the timing code a bit so that it can be used separate from the code which writes span exit messages see https://github.com/tokio-rs/tracing/pull/3063 and https://github.com/tokio-rs/tracing/pull/3038 which could be worth thinking about (note these are written for the yet to be released 0.2 version of tracing, so might not work without a few small mods for 0.1.x)

[timonv]

Nice. Expanding on this further, it would be kind of cool if the whole thing is a tracing layer 🤔 That's a win-win-win I think. Could provide some tooling around that to make it easier to use, while at the same time use it in otel and other tools.