zed icon indicating copy to clipboard operation
zed copied to clipboard

vector engine projection

Open mccanne opened this issue 2 years ago • 0 comments

The first step toward a working vector engine is simple projection. In this issue, we will wire the vector engine into the runtime so that it is actually invocable from a lake query. When all of the objects to be scanned have vectors (VNG files) and a projection is deemed worthwhile, the optimizer will rewrite the DAG to use a VecScan and push the projection into the scanner. VecScan will pull only the fields needed from the vcache and materialize the results into zbuf.Batches of serialized sequence data.

This will work only with vcache and lake data. zq running on VNG files will not be supported (yet).

mccanne avatar Oct 24 '23 15:10 mccanne