Inconsistent behaviour of configs and named_configs with ingredients

[thequilo]

I try to explain my question with the following example:

from sacred import Experiment, Ingredient

ing = Ingredient('ing')
ex = Experiment(ingredients=[ing])


@ing.config
def ing_config():
    a = 42


@ex.config
def config1(ing):
    b = ing['a'] * 3.1415


@ex.named_config
def named(ing):
    b = ing['a'] * 2


@ex.named_config
def named2():
    ing = dict(a=12345)


@ex.automain
def main(b):
    print(b)

Usual config scopes (defined with Ingredient.config) can depend on the config of sub-ingredients like config1 in the above example. This is not possible for named configs (named):

$ python test.py print_config
INFO - test - Running command 'print_config'
INFO - test - Started

Configuration (modified, added, typechanged, doc):
  b = 131.943
  seed = 635712928                   # the random seed for this experiment
  ing:
    a = 42
INFO - test - Completed after 0:00:00

$ python test.py print_config with named
Exception originated from within Sacred.
Traceback (most recent calls):
  File ".../sacred/config/custom_containers.py", line 141, in __getitem__
    raise KeyError(item)
KeyError: 'a'

But, named configs can modify the configuration of sub-ingredients which is not possible from within usual config scopes:

$ python test.py print_config with named2
INFO - test - Running command 'print_config'
INFO - test - Started
Configuration (modified, added, typechanged, doc):
  b = 38781.817500000005
  seed = 542129626                   # the random seed for this experiment
  ing:
    a = 12345
INFO - test - Completed after 0:00:00

Why don't ususal configs and named configs behave the same? Can we make both behave the in same way or include some option to change the behaviour for each config scope?

[Qwlouse]

Hi @thequilo , I know this is confusing, but it is intended behavior: named configs behave like updates, and not like normal configs (ideally I would have called them named_updates instead). The usecase for which they are intended is to store a certain set of parameters that make sense together. For example if I report multiple experiments in my paper, I like to have a named config for each, such that running them is easy. But for that they must be able to influence the config values of ingredients, so that you can for example set the dataset-ingredient and the network parameters in one named config.

Because of the way Sacred evaluates configurations, that means they have to be evaluated before the ingredient config is evaluated. So in your example ing does not have a value for a yet, when named is evaluated.

So what can be done? We could implement an actual named config, i.e. something that behaves like a normal configuration but that is optional. You can simulate that in your code by doing something along these lines:

@ex.config
def config1(ing):
   named_configs = []
    b = ing['a'] * 3.1415

@ex.config
def named(named_configs, ing):
    if 'named' in named_configs:
        b = ing['a'] * 2

and then run with

$ python test.py print_config with "named_configs=['named']"

I like the idea of having a feature like this built-in, but to avoid increasing the amount of confusion should probably include renaming the current named_config to named_update or so. I'll put it on my list, but since it would break the API, it won't happen before the next major release, which I'm afraid is not going to happen this year.

Does this clarify things for you? What is your opinion about this?

[thequilo]

Yes, this totally makes sense now! I'll for now try to use your suggested workaround.

I have two additional suggestions on what can be done:

1. Add a flag (as ignore in #339)

I think this is preferrable over the renmaing to named_update as it would not break the current interface. I didn't come up with a nice name for this yet

@ex.named_config(flag_that_executes_this_as_config_scope=True)
def named(ing):
    b = ing['a'] * 2

2. Analyze the dependencies

This one clearly is difficult to implement and may cause subtle unwanted behaviour in some cases. But, we could execute any named config scope as a "named config update" as long as it does not have an ingredient in its parameter list and otherwise execute it as a usual config sope (in toplogical order).