UX-friendly Configurability for Page Demoting/Moving

[klihub]

I think it would make more sense from the UX/user point of view to have the real internally used configuration parameters determined from a set of QoS-oriented user-visible configuration parameters.

I'm thinking/talking about things like these:

1. allowed maximum total memory BW consumption induced by- the demoter
2. allowed maximum per-NUMA node memory BW consumption induced by the demoter
3. allowed workload disruption/agressiveness
4. disruption/aggressiveness could be expressed for instance by something like how big percentage of the granted CPU time the container would/could lose as a result of the kernel extra-blocking the container because of pages being moved as requested by the demoter.
5. dynamic/self-adjusted control: Ideally 1. and 2. could/should be dynamically adjusted, if we could (reliably) measure the overall and per-controller utilization of memory buses/controllers and also somehow estimate the related UPI utilization/saturation levels from this. As long as there is spare BW and we're not violating per-container demoter QoS-restrictions we would pump up the volume, otherwise ease off for a while.
6. per-container allowed aggresiveness: the possibility to override 3. above on a per-container basis using annotations