iggy
iggy copied to clipboard
Write testcases for message cache
In iggy-server
messages cache works in per partition manner. When partition object is created, and cache is enabled, it creates internal SmartCache
object, which is essentially wrapper around VecDeque<Arc<Messages>>
. In other words, this is variable length ring buffer.
Messages in cache have to be coherent and sequential in terms of offset.
Cache has global (for whole server executable, for all partitions) usage counter, of which maximum value (in bytes) is interpreted from config. It's accessible via CacheMemoryTracker
. When message is received by server, it always ends up in cache. If cache is full, eviction happens.
At the server startup, it loads messages from disk (if any data is present) to cache of each partition proportionally:
In terms of execution, it looks something like:
Same formula is used at runtime, when cache is evicted. (proportionally from each partition)
During cache eviction memory usage looks something like:
Purpose of this task is to write a testcase, which would check if cache works fine.
Suggested OK scenario:
- initialize
System
usingTestSetup
(see integration/tests/streaming/topic_messages.rs) with enabled cache - append some messages
- see if cache has these messages
- append some more, above cache limit
- see if cache has evicted part of messages and cache size is below limit
Suggested negative scenario:
- initialize
SmartCache
with some bad messages (incoherent cache, non sequential messages offsets) - initialize
System
usingTestSetup
and aboveSmartCache
- append some messages
- see if cache has emptied because cache coherency check failed
These are just general suggestions.