Maintain memory usage counters when WriteBufferManager is disabled
WriteBufferManager (WBM) maintains 2 counters that track memtable memory usage by its clients (memory_used_ and memory_active_). These counters are std::atomic variables.
RocksDB's WBM may be enabled (runs with a positive buffer size) or disabled (no buffer size == 0).
When the WBM is disabled, it doesn't maintain the memory_active_ counter at all, and maintains the memory_used_ counter only if it runs with cache costing enabled. This is to save an expensive read-modify-write of atomic variables when they are not needed (WBM is disabled). In addition, it is possible to modify the WBM's buffer size, potentially causing it to switch from enabled to disabled or vice versa.
The WBM's counters are used to control the flushing and stalling mechanisms that the WBM is associated with. These mechanisms rely on the continuous maintenance of these values of these counters when the WBM is enabled.
The problem is that if the WBM runs in disabled mode (not updating one or both counters), and is then enabled, the flush and stall associated with WBM may cause serious system problems (e.g, continuous flushing, complete write stalling).
Let's consider the following scenario:
- The WBM is initially disabled and no cache is provided => both counters remain 0.
- The WBM is enabled
- A memtable is freed.
At step 3 above, WriteBufferManager::FreeMem() will be called, the memory_used_ counter will wrap around to a huge value and writes will be stalled indefinitely.