HomeStore icon indicating copy to clipboard operation
HomeStore copied to clipboard

Add long running log store tests.

Open sanebay opened this issue 9 months ago • 0 comments

Changes.

  1. Fix the truncation in corner cases.
  2. Add long running test for logstore. Stripped down version of test_log_store. Add burst of requests to all logstores, truncate, restart, add/remove logstores logdev's, ad rollbacks etc.
  3. Enable logstore test except the parallel truncation UT.

More details. Journalvdev maintains a list of chunks to store all the log entries. All append log entries are appended to the last chunk in the list(right side/tail offset) and truncate is applied to the head of the chunk list(left part/ data start offset). Whenever we append log entries, if we dont have enough space, we create a new chunk and append to the list. So log groups(batch of log entries) dont go across chunks. So there will be holes in these chunks at the end which are marked by end_of_chunk in chunk private data. The hole lies between end_of_chunk and (chunk_start + chunk_size) . When we read and we reach this hole, there is no data so we skip and move to the next chunk. Similarly if truncate happens to be in that hole also, we release the whole chunk and move to the next chunk. Also we set the data_start_offset to the start of the next chunk.

sanebay avatar May 13 '24 23:05 sanebay