MLOS icon indicating copy to clipboard operation
MLOS copied to clipboard

mlos_bench storage: track config actually run and other basic system metadata

Open bpkroth opened this issue 1 year ago • 2 comments

Some systems have constraints that our optimizers don't currently model (e.g. param A must be a linear combination of param B). When set to something else, they actually run with a nearby config.

Registering the "nearby" config when space adapter features from mlos_core, like LlamaTune, are enabled is problematic because the random project doesn't necessarily have an inverse from the new config point in the higher dimensional space that was actually used, back to a point in the lower dimensional sampled space the optimizer is using. Moreover, some optimizers expect you to register only points we've actually sampled.

Ideally, we'd have a way to model that in the optimizer's acquisition function so it only suggests values that are possible to use.

In the meantime, another option we could have is to simply track the parameters actually used by the system after a reconfig event.

To do this we might want to add a "get_current_config" operation script for each environment.

bpkroth avatar Jun 20 '23 19:06 bpkroth

Additional things that would be good to capture by default:

  • OS version
  • VM specs (CPU type, nominal speed, core count, geometry, RAM size, etc.)

bpkroth avatar Jul 31 '23 21:07 bpkroth

Related: https://thehftguy.com/2023/11/14/the-linux-kernel-has-been-accidentally-hardcoded-to-a-maximum-of-8-cores-for-nearly-20-years/

bpkroth avatar Nov 15 '23 17:11 bpkroth