Key-value lookups in Zed runtime

[mccanne]

We should have a way to issue queries to a sub-pool to populate a map literal that can be referenced in the Zed runtime... e.g., we will need syntax to declare the map's name (like a const) and configure the sub-query used to get a key/value. If the table is small, the entire contents could be read into memory. Otherwise, LRU replacement should evict entries when the table gets too big. To make this efficient, the pool-key of the map contents should be the key of the map. The syntax might look like this...

map table = from srcPool | configKey==$key | cut this:={key:configKey,value:interestingValue}

A reference to

... table[x] ... 

would then evaluate 'x' as the $key parameter to the subquery and load that key in the table. Once loaded, the sub-query need not run again.

The generality here would give full range over how much to load in for a given key reference, e.g., loading the entire pool contents.

Keys that are reference but not found will be set to nil (as the sub-query would return an empty result).

The object that implements the sub-query should also be able to listen for commits to the pool holding the map data and flush the map when changes are made. Obviously this wouldn't work well for quickly changing data, in which case, join should be used. This is for slowly changing state where cached key lookups will be more efficient than joins.

In a future issue, we can generalize this mechanism to do key lookups from external systems like reddis or SQL DBs.

[philrz]

#4141 was recently completed to provide an initial take at creating such maps. This issue can remain open to track the addition of some of the more sophisticated ideas proposed above, such as having such a map be updated in the runtime when more elements are loaded into its pool.

[philrz]

A community zync user recently expressed the need for this functionality. In their own words:

Would it be possible to dynamically define a map const by using the map aggregator function. Ex: const mymap = (from mymappool | map(|{key,value}|)). Such that, the const would become available throughout the entire following graph ? We have a workaround simply copy/pasting the entire source map in ZSON into the const definition, but some of the maps we are dealing with are very large and their content can change dynamically.

I should say fairly large, and not very large. In the order of a few hundreds rows, so not so large for Zed but large enough to make the copy/paste inconvenient and error prone.