flux-framework
Need a more user-friendly configuration scheme for qmanager
From PR #649:
As we discussed before, I'm not entirely happy with the current way of configuring queues. But I thought there is a benefit of getting the runtime logic correct as soon as possible and then improve upon user friendliness later.
To be more specific: it was much easier to implement and reason about multi-queue specifications when the configuration file and module load options were identical (current implementation). Once we make sure everything is correct and performant under the current implementation, ways to specify multi-queue can diverge.
For example, configuration file become more user friendly while load option specification stays the same.
For configuration file, I am thinking along the line of:
[qmanager]
# default if per-queue doesn't specify
queue-policy = "fcfs"
queue-params = "queue-depth=512"
[batch]
queue-policy = "conservative"
[debug]
queue-depth = 32
[all]
Internally, once the toml file is parsed, qmanager_opts would transform this form into the module load option form to process. This way, there is only one logic that provides a semantics for both conf file and load options.
Doing all of this at the same time focusing on the core multi-queue support was too time consuming and involved. (Too big of change as a single PR.)
Internally, once the toml file is parsed, qmanager_opts would transform this form into the module load option form to process. This way, there is only one logic that provides a semantics for both conf file and load options.
Config file looks nice to me. In case you might want future sub-tables below qmanager, it may be wise to add queue configurations to their own table queue? So you have qmanager.queue.batch. Just a suggestion at this point.
Generally, it would be nice if the core-provided flux_conf_t interface could help out with precedence and parsing load-time options. Using the config file above as an example, it would be nice to be able to use a dotted notation to override or add any of the supported keys from the TOML config, e.g.
flux module load qmanager qmanager.queue.batch.queue_depth=16
During module load and in the config reload handler, modules could first register default values, then update defaults from the broker provided conf_t, and finally override with load parameters. In the case of a config reload, there are no commandline parameters, so a config reload would override any past load-time paramters.
An interface for setting defaults and overriding with both broker-provided config and module load parameters could be provided by the libflux flux_conf_t interface, thus ensuring all modules behave in a similar manner.
This would have several nice features:
- reduce duplication of module config and parameter handling between modules
- unify module load-time parameter handling
- depending on how defaults are set, modules could advertise parameters they support, and these could be queried from
flux module infoor something. - any single configuration parameter can be overridden on the command line
- could probably easily be extended to support a standard rpc to query/set module parameters
I agree there isn't time to develop this now.
I apologize if my long-winded description is just restating what you said above, but wanted to make sure I have the right idea.
Config file looks nice to me. In case you might want future sub-tables below qmanager, it may be wise to add queue configurations to their own table queue? So you have qmanager.queue.batch. Just a suggestion at this point.
Like it! This looks far more extendable!
Generally, it would be nice if the core-provided flux_conf_t interface could help out with precedence and parsing load-time options. Using the config file above as an example, it would be nice to be able to use a dotted notation to override or add any of the supported keys from the TOML config, e.g.
flux module load qmanager qmanager.queue.batch.queue_depth=16
Yeah this will give you a nice hierarchical way to override the config file parameters at load time.
When you don't have a configuration file to begin with, however, how do you envision we can specify low level subtable keys like multiple queues in this case?
flux module load qmanager qmanager.queue=batch,debug qmanager.queue.batch.queue_depth=16
I didn't have a real good idea there and punted with an ad hoc specification like
flux module load qmanager queues="batch debug" queue-params-per-queue="batch:... debug:..."
BTW, one thing that I needed was to provide a upper option manager a way to effect which options need to be processed first. You want
flux module load qmanager queue-params-per-queue="batch:... debug:..." queues="batch debug"
also equally works fine.
I added support in https://github.com/flux-framework/flux-sched/pull/649/commits/58bb1ce2bdf0ef0c9a06e1859d719de1a430af5d
In case this lession is useful for flux-core level generic config/module management layer.
When you don't have a configuration file to begin with, however, how do you envision we can specify low level subtable keys like multiple queues in this case?
This is tricky, but maybe it could be handled like KVS where keys are created if they do not exist when they are first set? So something like this would "just work"
flux module load qmanager queue.batch.queue_depth=16 queue.debug={}
?
However, I agree with @garlick, that likely setting complex configurations on the commandline isn't an initial (or important, or perhaps even useful?) use case for now, so maybe some things are just not supported?
However, I agree with @garlick, that likely setting complex configurations on the commandline isn't an initial (or important, or perhaps even useful?) use case for now, so maybe some things are just not supported?
Should the proposal on our multi-source option specification model be: a load option set shall be limited to a subset of the corresponding config option set? As I mentioned at one of our recent coffee hour meetings, this should work as far as we are clear on what this subset is and semantics.
setting complex configurations
Although in this case I agree there isn't compelling use cases for specifying multiple queues as part of load options, I can see other cases where you want to specify sub-table parameters. Even the original qmanager.toml introduces sub-tables. If your config time specification is hierarchal like that, you want to be able to specify it at load time even when there is not configuration file?
A question: Speaking of a multi-source option specification model, how do you see this in relation to compile time option value specification?
Right now I use the same technique for all three sources: compile time, config file time and load time.
depending on how defaults are set, modules could advertise parameters they support, and these could be queried from flux module info or something.
I meant to comment that this is compelling and rids each module provider of having to write its own front end!
Could this also be a path for generic module parameter control?
flux module param-get qmanager.... vs. flux module param-set qmanager.queue...