MLOS
MLOS copied to clipboard
mlos_bench storage: track config actually run and other basic system metadata
Some systems have constraints that our optimizers don't currently model (e.g. param A must be a linear combination of param B). When set to something else, they actually run with a nearby config.
Registering the "nearby" config when space adapter features from mlos_core, like LlamaTune, are enabled is problematic because the random project doesn't necessarily have an inverse from the new config point in the higher dimensional space that was actually used, back to a point in the lower dimensional sampled space the optimizer is using. Moreover, some optimizers expect you to register only points we've actually sampled.
Ideally, we'd have a way to model that in the optimizer's acquisition function so it only suggests values that are possible to use.
In the meantime, another option we could have is to simply track the parameters actually used by the system after a reconfig event.
To do this we might want to add a "get_current_config" operation script for each environment.
Additional things that would be good to capture by default:
- OS version
- VM specs (CPU type, nominal speed, core count, geometry, RAM size, etc.)
Related: https://thehftguy.com/2023/11/14/the-linux-kernel-has-been-accidentally-hardcoded-to-a-maximum-of-8-cores-for-nearly-20-years/