omegaconf
omegaconf copied to clipboard
omry
allow exporting to dotlist-style format
It would also be useful to be able to export in a dotlist-style format. I've partially implemented this for my own projects, as it's critical when wanting to serialize a config for usage in a system where nested hash maps are not allowed.
In particular, HDF5 group attributes can only map strings to booleans, strings, numerical values, or arrays (of numerical types). I can imagine this would also be useful in other contexts, like when wanting to read a config in a strongly typed language.
I've partially implemented this in PR #518 and would be happy to implement the rest. I think a reasonable set of features to include would be:
- allow export of
DictConfigto a flattened dict - create configs from flattened dicts, in a manner similar to creating from dotlists
That's not a bad idea. After seeing your initial implementation I whipped together something a bit more concise and with a bit more testing (that also supports lists, and does not rely on to_container but operate directly on the config object, so it's probably also a bit faster). My function returns a List[Tuple(str, Any)], not a Dict. I think it's closer in nature to dotlist and can easily be converted to a dict or to a proper dotlist.
Roundtrip is not always possible, for example {"a": {}} cannot be reconstructed unless we use some kind of sentinel for an empty dict. same goes for lists.
I will put up my PR for discussion.
See #520
Round trip is also not possible when lists are involved (from_dotlist) is always returning a dict. In practice this is probably good enough but keep in mind that this is not fool proof serialization.
I am actually not convinced I want to officially support the reconstruction because it's doing such a poor job at reconstructing configs in many cases.
Roundtrip is not always possible, for example {"a": {}} cannot be reconstructed unless we use some kind of sentinel for an empty dict. same goes for lists.
I think this is doable, we would just have to check for an empty container before recursing. If the container is empty, add ('a', dict()) to the list. if its not empty, recurse. Is that what you meant by sentinel?
I am actually not convinced I want to officially support the reconstruction because it's doing such a poor job at reconstructing configs in many cases.
How is it failing? (Sorry if this information is available somewhere, I'm new to this repo) Does the round-trip failing have something to do with conversion to and from strings when putting into the dotlist format?
How about you check-out my PR and try to add some tests? specifically, you will see that a list is being converted to a dict.
There could be ways to encode this information somehow but I don't know if it's worth the extra complexity. There is this entire other area of Structured Configs which will not be decoded properly (but that's a problem with actual yaml serialization as well).
At this point I do not plan to invest additional time into this feature before 2.1 is released. If you care about it feel free to pick up my PR and complete it. (discussion, additional tests as needed and documentation updates).
I would also find this feature useful. Having it would allow, e.g., to easily dump the configs into MLflow Tracking for subsequent comparison of experiments. Within this scope, it does not matter whether the returned format is a dotlist or a flat dict, with the latter being slightly more convenient.
It would be better if mlflow supported hierarchical configs. have you suggested it to them?
Is there any update on this?
Would this with
separator='.'be a first start to such a solution?
This is unlikely to make it into OmegaConf in the short term unless someone submits a PR. It’s best you implement your own function to do this if you need it now.
