[Feature] Support OmegaConf-style resolvers in configs

[collinmccarthy]

What is the feature?

I'm working on a project that requires me to try many combinations of models, datasets, tasks and augmentation pipelines. You can do this pretty well without manually defining hundreds of config files using the "${key}" syntax. But one big challenge remains when you have a parameter that is a function of a "${key}" value. For example:

Normally we would define things in the same file and reference them directly, e.g.

img_scale = (640,480)

train_pipeline = [
    dict(
        type=mmcv.transforms.RandomResize,
        scale=(2 * img_scale[0], 2* img_scale[1]),
    ),
    dict(
        type=mmdet.datasets.transforms.Pad,
        size=img_scale
    )
]

Now suppose we want to use the same pipeline but with many different image sizes (each coming from different datasets). Then we will separately pass in the two configs with something like --dataset_cfg and --aug_cfg, so we never have to define a single config file with all combinations of datasets and augmentations. In the aug_cfg file the "${key}" syntax would work for the Pad augmentation, but not for the RandomResize.

train_pipeline = [
    dict(
        type=mmcv.transforms.RandomResize,
        scale=(2 * "${img_scale}"[0], 2* "${img_scale}"[1]),   # Syntax error
    ),
    dict(
        type=mmdet.datasets.transforms.Pad,
        size="${img_scale}",  # Works
        pad_val=dict(img=(114, 114, 114))
    )
]

The only solutions (discussed below) are to change the augmentation class or support lambdas or OmgeaConf-style "resolvers" somehow.

Any other context?

The best workaround I've found for something like RandomResize is to create a subclass like CustomRandomResize as follows.

class CustomRandomResize(RandomResize):
    def __init__(self, *args, size, scaling_factor, **kwargs):
        scale = (size[0] * scaling_factor, size[1] * scaling_factor)
        super().__init__(*args, scale=scale, **kwargs)

Then we can update the config file to just be

[
    dict(
        type=CustomRandomResize,
        size="${img_scale}",
        scaling_factor="${scaling_factor}"
    ),
    dict(
        type=mmdet.datasets.transforms.Pad,
        size="${img_scale}",
        pad_val=dict(img=(114, 114, 114))
    )
]

But IMO this shouldn't be necessary to modify the APIs of potentially many functions just to handle config-file limitations. I would like to propose a real solution with more config-file flexibility. I understand there may be security risks with supporting functions or lambdas in config files, so we could add an unsafe flag with a default of False like Detectron2 does, along with a warning of some kind.

This type of thing is currently supported with OmegaConf resolvers and it works really well. And with the pure Python-based config files, it seems more doable than before. I'm not sure how complicated the OmgeaConf resolver implementation is, and I'm not suggesting we support everything they do, but here are a couple of thoughts.

1. Somehow support calling functions such as OmegaConf.register_new_resolver() and OmegaConf.create(), maybe by determining which nodes are OmegaConf config nodes, then parsing them separately and combining the config fields afterwards. So the config object would (optionally) have OmgeaConf config nodes after reading in the file. It could even be as easy as modifying the config-file compilation process to allow these functions to be called, but nothing else.

2. Support a custom syntax for lambdas with nested ${keys}. So something like scale="${lambda: (2 * ${img_scale}[0], 2 * ${img_scale}[1])}$ maybe. I think this would be re-inventing the wheel w.r.t. OmegaConf resolvers, but maybe we could restrict it to scalar "${key}" values if that makes it easier, e.g. scale="${lambda: (2 * ${img_scale_h}, 2 * ${img_scale_w})}$. Unless this is a super easy solution I think the first idea is better (more maintainable).

Thank you for the consideration! And just FYI I don't see myself having time to work on a PR here anytime soon (maybe June 2024 at the earliest), so I apologize in advance for that.

[HAOCHENYE]

I apologize for the delay in my response and I appreciate your suggestions. However, there are a few points that are causing some confusion for me.

1. Is the syntax "${key}" you mentioned intended to refer to variables defined in the base config? Currently, MMEngine supports two style configs: the text style config (the older one) and the Python style config. If you are using the text style config, you can reference base variables using _base_.key. On the other hand, if you are using the Python style config, you can simply refer to the base variable as img_scale because it is inherited through from {base_config} import *. You have the flexibility to access any base variables using valid Python syntax and import rules.

2. Are you suggesting the definition of functions (including lambda functions) within the config file? We have actually considered this option. The primary reason we haven't supported it thus far is that it's challenging to ensure the reliability of the dumped config when functions are involved. In the upcoming version of MMEngine, we plan to introduce support for defining functions in the config file. However, we cannot guarantee that the dumped config will be fully reusable.

[hiyyg]

The new python style config does not support recursively instantiation without a registry. However, it is already supported by using omegaconf.

[collinmccarthy]

Wow, I'm so sorry I thought I responded to this, but clearly I did not.

@hiyyg I am using a registry even with the python-style config, so I'm just looking for lamba functions in particular. But I can see the benefit from not having to use a registry in general.

@HAOCHENYE

Is the syntax "${key}" you mentioned intended to refer to variables defined in the base config? Currently, MMEngine supports two style configs: the text style config (the older one) and the Python style config. If you are using the text style config, you can reference base variables using base.key. On the other hand, if you are using the Python style config, you can simply refer to the base variable as img_scale because it is inherited through from {base_config} import *. You have the flexibility to access any base variables using valid Python syntax and import rules.

I didn't realize this at the time, but what I'm referring to is a syntax supported in mmdetection, not mmengine directly: mmdet.utils.replace_cfg_vals.py. It turns out that it isn't sufficient to do it just like they are, since one "${key}" can reference another "${key}", so you have to resolve it multiple times. Here's a rough snippet of how I actually use it.

# Replace until all ${key} are replaced
# Same pattern of string "${key}" as in mmdet.utils.replace_cfg_vals()
regex = r"\$\{[a-zA-Z\d_.]*\}"

done = False
while not done:
    # Force lazy mode since mmdet.utils.replace_cfg_vals uses `=`
    #   which triggers objects to be built
    config = config._to_lazy_dict()
    matches = re.findall(regex, config.values())
    if len(matches) == 0:
        done = True

return config

(This by itself may not work, what I got working is a bit more complicated and I've left some stuff out for readability. It's just an example.)

Are you suggesting the definition of functions (including lambda functions) within the config file? We have actually considered this option. The primary reason we haven't supported it thus far is that it's challenging to ensure the reliability of the dumped config when functions are involved. In the upcoming version of MMEngine, we plan to introduce support for defining functions in the config file. However, we cannot guarantee that the dumped config will be fully reusable.

Yes! This is exactly what I want. I'm not sure how it works in OmegaConf (if they support the "dump" functionality w/ reloading at all), but what I'm looking for is the ability to create a lambda function just like they do with their "Custom Resolvers". For my example above, maybe it would look like this.

mmengine.register_new_resolver(
    "random_resize", 
    lambda resize_factor, img_scale: (resize_factor * img_scale[0], resize_factor * img_scale[1])
)

resize_factor = 2
train_pipeline = [
    dict(
        type=mmcv.transforms.RandomResize,
        scale="${random_resize: ${resize_factor}, ${img_scale}}"  # Omega-Conf style resolver
    ),
    dict(
        type=mmdet.datasets.transforms.Pad,
        size="${img_scale}",  # Works
        pad_val=dict(img=(114, 114, 114))
    )
]

I'm using the syntax from OmegaConf: Custom resolvers. For what I'm trying to do this would be incredibly useful, and I'm sure over time others will agree it is helpful for them as well. Thank you for the consideration!

[HAOCHENYE]

The new python style config does not support recursively instantiation without a registry. However, it is already supported by using omegaconf.

Actually, you can use Python-style config without a registry by defining a custom build function. I'm curious about the point that user can initiate instance without defining any build function (registry could be considered a wrapper of build function). Could you give me an example that how omegaconf? I remember that Detectron2 still need to use a decorator to handle the instantiation

[hiyyg]

The new python style config does not support recursively instantiation without a registry. However, it is already supported by using omegaconf.

Actually, you can use Python-style config without a registry by defining a custom build function. I'm curious about the point that user can initiate instance without defining any build function (registry could be considered a wrapper of build function). Could you give me an example that how omegaconf? I remember that Detectron2 still need to use a decorator to handle the instantiation

It is not trivial to recursively instatiate objs with the python style config. We can achieve this by convert the new python-style config to d2's omegaconf, but it would require some tricky conversions.

D2's lazy config can recursively instantiate any callable obj without any decorators.

Maybe you are talking about the old-style yacs config.

[HAOCHENYE]

The new python style config does not support recursively instantiation without a registry. However, it is already supported by using omegaconf.

Actually, you can use Python-style config without a registry by defining a custom build function. I'm curious about the point that user can initiate instance without defining any build function (registry could be considered a wrapper of build function). Could you give me an example that how omegaconf? I remember that Detectron2 still need to use a decorator to handle the instantiation

It is not trivial to recursively instatiate objs with the python style config. We can achieve this by convert the new python-style config to d2's omegaconf, but it would require some tricky conversions.

D2's lazy config can recursively instantiate any callable obj without any decorators.

Maybe you are talking about the old-style yacs config.

Ah, I get what you're saying. However, I believe that the attribute of not relying on the build function can be attributed more to D2's lazy config than to omegaconf.

Considering the programming habits of the OpenMMLab series of repositories, we have chosen a method different from LazyConfig. LazyConfig requires a function to be called at the training entrypoint to build all instances, whereas the current implementation in OpenMMLab leans more towards having the parent module manage the construction of its child modules. In other words, when and where to build a specific module is determined by the code. These two distinct programming styles mean that the python style config still needs to rely on the build function.

[collinmccarthy]

I ended up implementing this in a similar way to the "${key}" evaluation above. I can use something like the following the config file:

num_classes = 80
num_det_layers = 3
loss_weight = "{{eval: ${num_classes} / 80 * 3 / ${num_det_layers} }}"

First I replace "${num_classes}" and "${num_det_layers}" with "80" and "3" respectively, which gives me loss_weight = "{{eval: 80 / 80 * 3 / 3 }}" after the first pass. Then I use pattern = r"\{\{eval:(.+)\}\}" to extract the first group " 80 / 80 * 3 / 3 " and pass this to eval() to get 1.0, which gives loss_weight = 1.0. If you have an eval str which references another eval str you can just iteratively evaluate the strings from innermost to outermost (with a slightly more complicated regex pattern) to handle nested eval strings.

This approach works for more complicated expressions too:

image_wh = (640, 480)
scale_img = "{{eval: (${image_wh}[0], 2*(${image_wh}[1])) }}"

This will store scale_img = (640, 960). Overall for what I'm trying to do this works great. I would love to see something similar incorporated into mmengine so people can start using this to duplicate less code in config files.

[HAOCHENYE]

I apologize for the delayed response. I've been trying to find a clear and Pythonic solution to address this issue since understanding your perspective. The crux of the matter is that immutable variables cannot be altered by changing their reference. In your scenario, if both scale_img and image_wh are mutable:

image_wh = [[640, 480], [640, 480]]
scale_img = [image_wh[0], image_wh[1]]

Simply modifying image_wh would suffice. However, if scale_img is a tuple of integers, altering its reference in image_wh won't affect the value of scale_img. Hence, we need a Resolver and a string format syntax designed for lazy initialization of scale_img until it accepts the value provided by the user.

While this approach can address the issue, I believe that since we're aiming for a Pythonic configuration, the solution should be more intuitive and explicit. The string-style syntax might be too complex for most users.

Google's ml_collection offers an interesting perspective. Consider defining the config as follows:

from mmengine.config import PlaceHolder

num_classes = PlaceHolder(80)
num_det_layers = PlaceHolder(3)
loss_weight = num_classes / 80 * 3 / num_det_layers

image_wh = PlaceHolder((416, 416))
scale_img = (image_wh[0], image_wh[1] * 2)

Subsequently, when we load this config:

cfg = Config.fromfile('cfg path')
assert cfg.scale_img  == (416, 832)
cfg.image_wh = (256, 256)
assert cfg.scale_img  == (256, 512)

This demonstrates that the PlaceHolder variable only logs the value and operations. Its value is computed only upon accessing the corresponding field. We can also modify the source value of the Placeholder, and this change will be reflected across all Placeholders since the operation is only utilized to compute the actual value when accessed via cfg.

[collinmccarthy]

Thank you for following up. I understand the string syntax could be too complex for most users. The other issue is that it's easy to make a mistake, and debugging things can be quite challenging at first.

This PlaceHolder solution looks great! I can't think of any situations where it wouldn't work for my use case, so I'd be thrilled to see it incorporated in the near future.