brimdata
first bits of vector cache for ZST objects
This commit provides the first most basic implementation of the vector cache for ZST objects. The new "zed dev vcache" command is added to support some ztests for exercising the cache. It is not yet wired into the query planner.
Currently, this code represents the "slow path" of the vector cache, which supports only simple projections of ZNG-seralized values.
An object is loaded into the cache by reading the ZST metadata and building a hierarchical data structure of the Zed types implemented in package vcache. Each of the types implements the vcache.Vector interface. Currently, the only method on Vector is NewIter() to create an iterator to enumerate the values represented by each vector. In future PRs, we will introduce many more methods on this inteface to push down searches, boolean logic, aggregate functions, and so forth. We will also introduce Go-native vectors at some point too so that the zcode deserialization does not have to be done when pushing down operations.
Vector data is not loaded into the cache until it is referenced. This way a pattern of queries that operates on a limited set of fields will only load the needed fields into the cache. The loading happens on demand right now when an iterator is created and care has been taken to perform all the vector loads from storage in parallel as to avoid sequential round-trips to slower storage layers like S3.
There is also a simple implementation of top-level field projections. In this case, a projection can be created from a list of top-level fields. This differs from the cut operator a bit as error("missing") are omitted, which should be okay for how we plan to wire this into the planner. Also, projections of dotted field references are not yet supported.
Currently, each object leaves its file open forever and memory is never freed from the cache. In subsequent PRs, we will add a mechanism to age and close the files as well as LRU replacement policy to the cache with some sort of memory limit.
