Inspect the output at a midpoint in a Zed script

[philrz]

A recent feature idea from @jameskerr:

I’ve been wanting an operator that breaks early from a pipeline for debugging. It would be the equivalent of commenting out all the lines below it. I want to be able to see the intermediate forms the data takes before the final output. It could be called inspect, break, bypass, debug… I like inspect. It’s like you are inspecting the script at that point. An example:

from pool
| where id="100"
| collect(array_thing)
| inspect // <---- ends the query right here
| over this => yield {a: 1, b: 2, thing: this}

I can easily move that inspect up and down in the pipeline to see the steps one took to reach the end result.

When discussing as a team, it was pointed out that this would be tricky to use as shown here if the inspect appeared on a branch of a forked pipeline. An idea proposed to deal with this would be a general "named outputs" feature such that inspect could be used in multiple locations and each invocation marked with some unique identifier. That unique identifier could then be used to differentiate the different output streams. For instance, the Zui app could pop up different panels/windows that show the different outputs.

All this said, since single-branch pipelines are currently quite common, it seems nothing would preclude us from writing an initial version similar to the original proposal and just documenting that it's only intended for use in single-branch pipelines.

[philrz]

One of our community users recently went through some join struggles in a Slack thread and I mentioned this issue to them in a closing remark after they described having chopped up their pipeline into individual pieces during debug so they could see what the output looked like at certain midpoints. Here was their feedback on the idea:

Oooo I really like that - will simplify debugging - is that common in other query languages? It'll be awesome if you dumped the results of inspect in a separate pool or something similar

(@mccanne later noted that jq has debug, for example.)