Pipeline profiler

[countvajhula]

Racket includes a statistical profiler that can be useful for profiling in general. This profiler produces reports containing measured aggregate times for various functions, along with the proportion of time spent in each function, and allows you to sort the report in terms of these columns in the data. This data can be useful to identify performance bottlenecks and avenues for improvement.

Yet, the data isn't especially easy to decipher, and takes experience and some art. For Qi, it would be useful, in addition, to have a "pipeline" or "structural" profiler, which measures the time spent in each flow component. For instance, in a flow like:

(~> sqr (-< add1 sub1) *)

There are many flows of interest here -- the overall flow, each of its components, and each component of those components, which come out to 1 + 3 + 2 = 6 flows. The pipeline profiler should report on the time spent in each of these flows, organized hierarchically. This could prove to be a more intuitive and accessible alternative to the usual statistical profiler, for users looking to make quick performance assessments.

A color-coded visualization (e.g. red = "clogged", green or blue = "free flowing") built on top of the underlying performance profile data would be ideal.

Possible implementations: A. Implement the profiler as a macro which rewrites component forms, e.g. (profile (~> sqr (-< add1 sub1) *)). It may not be straightforward to rewrite nested flows, though, e.g. the contents of -<. B. Implement this as a Qi dialect, with forms like -< and ~> overridden to be equivalent versions that also record performance data, possibly as side effects, or in a hash. This may be feasible once a "core language" has been distilled, but it seems like a rather heavyweight option. It may involve overriding every form in the core language.