zed icon indicating copy to clipboard operation
zed copied to clipboard

parallelize streams

Open mccanne opened this issue 2 years ago • 1 comments

For queries that are flow insensitive, the same parallelization logic that we use for lake can also apply to streams (e.g., zq or internal readers like ingest).

We should logic to the optimizer to handle this case. The same logic that applies to lister/scanner parallelization can easily adapt to kernel.Reader.

mccanne avatar Apr 23 '23 21:04 mccanne

#4920 replaced kernel.Reader with dag.DefaultScan, to which this issue still applies.

nwt avatar Dec 06 '23 15:12 nwt