iggy icon indicating copy to clipboard operation
iggy copied to clipboard

Write testcases for message cache

Open hubcio opened this issue 1 year ago • 2 comments

In iggy-server messages cache works in per partition manner. When partition object is created, and cache is enabled, it creates internal SmartCacheobject, which is essentially wrapper around VecDeque<Arc<Messages>>. In other words, this is variable length ring buffer.

Messages in cache have to be coherent and sequential in terms of offset.

Cache has global (for whole server executable, for all partitions) usage counter, of which maximum value (in bytes) is interpreted from config. It's accessible via CacheMemoryTracker. When message is received by server, it always ends up in cache. If cache is full, eviction happens.

At the server startup, it loads messages from disk (if any data is present) to cache of each partition proportionally: image

In terms of execution, it looks something like: image

Same formula is used at runtime, when cache is evicted. (proportionally from each partition) During cache eviction memory usage looks something like: image

Purpose of this task is to write a testcase, which would check if cache works fine.

Suggested OK scenario:

  • initialize System using TestSetup (see integration/tests/streaming/topic_messages.rs) with enabled cache
  • append some messages
  • see if cache has these messages
  • append some more, above cache limit
  • see if cache has evicted part of messages and cache size is below limit

Suggested negative scenario:

  • initialize SmartCache with some bad messages (incoherent cache, non sequential messages offsets)
  • initialize System using TestSetup and above SmartCache
  • append some messages
  • see if cache has emptied because cache coherency check failed

These are just general suggestions.

hubcio avatar Oct 10 '23 15:10 hubcio