Free-form auto-updating with custom keys

[EMCarrami]

Hi,

Was wondering if it may be possible to add a functionality that would allow the user to specify a base config from cli using a predfined flag (e.g. --config) and then allowing for nested update of keys within that config in a free-form manner such that not all keys need to be pre-defined.

The functionality can also be extended to allow for further config files to be added to specific keys in the original config.

The main use case is when many different objects need to be configured while all those objects are subject to change during the dev time (this is in particular very common in ML projects.)

Below is a simple code that I use in my project, based on argparse. It isn't robust to many potential issues but thought to add it for for context:

def simple_parser(config_key: str = "config") -> Tuple[Dict[str, Any], Dict[str, Any]]:
    """Parse config file and update args."""
    parser = argparse.ArgumentParser()
    _, args = parser.parse_known_args()
    args_dict = {}
    for i in range(0, len(args), 2):
        k, v = args[i], args[i + 1]
        assert k.startswith("--"), f"invalid key {k}"
        assert not v.startswith("--"), f"invalid value {v}"
        k = k.replace("--", "")
        assert k not in args_dict, f"{k} key is used more than once."
        args_dict[k] = v

    config = load_config(args_dict.pop(config_key))
    extra_kwargs = {}
    for k, v in args_dict.items():
        try:
            v = literal_eval(v)
        except ValueError:
            pass
        keys = k.split(".")
        end_key = keys.pop()
        _config = config
        for _k in keys:
            try:
                _config = _config[_k]
            except KeyError:
                raise KeyError("nested arg parsed without a corresponding key in the config")
        if end_key in _config:
            _config[end_key] = v
        else:
            extra_kwargs[k] = v
    return config, extra_kwargs

[tbooth]

I think that my proposal for easier subclassing of the parser classes would help with this. See example 3 in https://github.com/tbooth/ConfigArgParse/blob/devel/tests/test_config_parser_subclasses.py

However, I don't think this is going to be a candidate for a feature to add to the core library.