Enhancement: LRU cache for query results

[lukekim]

Goal-State/What/Result

In-memory LRU caching of query results

Example:

runtime:
  results_cache:
    enabled: true
    cache_max_size: 128MiB
    eviction_policy: lru 
    item_ttl: 1s

datasets:
   ...

Why/Purpose

To provide accelerated performance for federated and fallback queries (in conjunction with behavior on zero results) and to prevent many redundant queries within a small time period.

By When

A target date for done-done completion of the entire enhancement.

Issue/Spec written and reviewed: 05/20 ... Done-Done: 05/23

Done-Done

• [x] First Principles Driven
• [x] The Algorithm
• [ ] PM/Design Review
• [ ] UX Review
• [ ] Threat Model / Security Review
• [ ] Tests
• [ ] Documentation
• [ ] Samples/Quickstarts/Tutorials
• [ ] Release Notes

The Algorithm

• [x] Every requirement questioned?
• [x] Delete (Scope) any part you can.
• [x] Simplify.
• [x] Break down into smaller iterations/milestones.
• [x] Opportunities for automation.

Spec

• [x] Cache configuration

runtime:
  results_cache:
    enabled: true
    eviction_policy: lru 
    cache_max_size: 128mb
    item_ttl: 1s

• enabled - optional, true by default (if there is a non-empty results_cache section defined)
• cache_max_size - optional, maximum cache size. Default is 128mb
• item_ttl - optional, cache entry expiration time, 1 second by default. Duration format follows existing conventions in Spice
• eviction_policy - optional, specifies the eviction policy to be used when the cached data reaches the cache_max_size. The default and currently the only supported policy is 'lru'.
• [ ] New telemetry events are added:
  • results_cache_request_count
  • results_cache_hit_count
  • results_cache_size
  • results_cache_item_count
  • results_cache_max_size
• [ ] New cache header is added to Spice http query response: X-Cache: HIT/MISS (no etag, age for simplicity). Exists only when cache is enabled.
• [ ] Caching is enabled by default

Out of scope for LRU cache alpha

• Header/option to bypass Cache
• Cache eviction based on accelerated table refresh (data modification).
• Manual cache eviction (API/CLI)
• Cache hit/miss indication when using Spice REPL
• Cache size telemetry (num entries or memory size in mb)

How/MAP

• [x] #1422
• [x] #1423
• [x] #1424
• [x] #1425
• [ ] #1471
• [ ] #1457
• [ ] Implement/run benchmarking tests
• [ ] Documentation is updated with Spice In-memory caching support
• [ ] Quickstart has section related to Spice In-memory caching

Implementation (issues)

QA