xous-core
xous-core copied to clipboard
Refactor key fill/scan for PDDB
As the PDDB caches more entries, it doesn't have a mechanism to add backpressure to prevent memory exhaustion (there is already a way to flush cache entries, but there's no trigger to cause cache entries to go away).
Use IncreaseHeap
with an argument of 0 https://docs.rs/xous/latest/xous/syscall/enum.SysCall.html#returns-1 to determine the current heap size and cull cache entries accordingly to keep memory footprint within a pre-defined bound (that also needs to be defined).
- [x] define backpressure bound
- [x] implement backpressure in hosted mode
- [x] pass tests in hosted mode
- [x] test on hardware
example of how the IncreaseHeap syscall can be used, there isn't a sugared call around it yet:
https://github.com/betrusted-io/rust/blob/1e18471c47493efe426828beb75c27d60fdf59d1/library/dlmalloc-xous/src/xous.rs#L28
So in the process of implementing backpressure, I've reminded myself of another problem. The subject of this issue has been renamed to reflect that.
Namely, the KeySmallPool is written assuming that you get a "full scan" of all the keys -- because it needs to dynamically compute the remaining space available in each small key page as part of its bookkeeping.
As part of this process, we read in all the small key data. This will become a Bad Idea very quickly as we start to store more and more small keys. Right now, backpressure is written such that we can evict said data once it has been read in, but, it's only a temporary fix because the fill()
routine is called quite frequently to make sure that metadata is populated.
The new issues are:
- [ ] Figure out a better way to do reserved vs available space bookkeeping. As a reminder, the
reserved
space parameter is for small keys that give analloc_hint()
-- it's extra space held aside so that keys that have slightly variable sizes don't incur expensive allocation/deallocations. The per-poolavailable
space is the total amount of space left in the small key pool. Ideally, we come up with a way to compute theavailable
space and/or track it in a data structure so we can have more flexibility in partial reads of a dictionary. - [x] Don't read in all the data during a fill operation for metadata
- [x] Backfill the cases that would now fail because we assumed the data was available.
Above is more or less a notional list. This is a fairly major task, maybe it will take a couple days just to wrap my head around what it all means, and then a couple days of testing to make sure nothing broke.
Upgrading to bug because once backpressure builds up and the keys are flushed, a panic is triggered.
Fixed with f71278b422f4b6e47036b1dd265506786bab8afe