zed icon indicating copy to clipboard operation
zed copied to clipboard

Examples that use the lake

Open philrz opened this issue 1 year ago • 0 comments

Some docs examples populate a scratch SuperDB data lake so that lake-specific operations using loaded pool data can be shown further down the page.

Details

There's two pages that use a similar approach of prepping a scratch lake up top and then referencing it further down. One is for the from operator:

https://github.com/brimdata/super/blob/8e86ac0ff4c9da6c278eb0ec9a8efba9f888c9d5/docs/language/operators/from.md?plain=1#L81-L96 https://github.com/brimdata/super/blob/8e86ac0ff4c9da6c278eb0ec9a8efba9f888c9d5/docs/language/operators/from.md?plain=1#L144-L146

Another is for the load operator:

https://github.com/brimdata/super/blob/8e86ac0ff4c9da6c278eb0ec9a8efba9f888c9d5/docs/language/operators/load.md?plain=1#L37-L49 https://github.com/brimdata/super/blob/8e86ac0ff4c9da6c278eb0ec9a8efba9f888c9d5/docs/language/operators/load.md?plain=1#L62-L67

It should also be noted that the super db tutorial understandably makes heavy use of the lake but has never even tried to be mdtest-protected in the past. Browsing the doc again right now to refresh myself, it's probably because there's too many unique outputs (e.g., commit/pool IDs) that it would be clunky to make it mdtest-runnable while still being readable. So perhaps that one stays as it is no matter what.

Ideas

This one feels like it would be challenging. Perhaps it's a special case of #5631 in that it implies having some separate Linux-like environment where shell commands can run to create input/state and then having that wired up so the Playground example can call on it. I'm quick to admit limited knowledge here as to what's possible.

philrz avatar Feb 06 '25 20:02 philrz