Validate argument names

[ondrejbiza]

Hello, I found that if I call my script with incorrect argument names, no error is thrown.

E.g. calling python train.py with "eopchs=5" instead of python train.py with "epochs=5" will not result in any error, but the script will run with an incorrect number of epochs.

Is there any option to validate the arguments?

Thanks, Ondrej

[thequilo]

As long as you don't run the script with the --force / -f flag, sacred raises an error if you pass unknown (not set in a default or named config and not an argument of a captured function) arguments. Are you sure that you don't have the same typo somewhere in your script?

A quick example:

from sacred import Experiment

ex = Experiment('test')

@ex.config
def config():
    epochs = 1

@ex.automain
def main(epochs):
    print(f'Running for {epochs} epochs')

$ python test.py with eopchs=5
sacred.utils.ConfigAddedError: Added new config entry that is not used anywhere
Conflicting configuration values:
  eopchs=5

[ondrejbiza]

I managed to replicate the error in this minimal example. It happens when I add two different config files:

config.json

{ "epochs": 2 }

config2.json

{ "learning_rate": 0.01 }

test.py

from sacred import Experiment

ex = Experiment('test')
ex.add_config('config.json')
ex.add_config('config2.json')

@ex.automain
def main(epochs, learning_rate):
    print(f'Running for {epochs} epochs')

Now python test.py with "eopchs=5" runs without any errors.

[thequilo]

This looks like a bug. Thanks for reporting!

I got down to:

from sacred import Experiment

ex = Experiment('test')

ex.add_config({})
ex.add_config({})

@ex.automain
def main():
    pass

$ python test.py with eopchs=5
WARNING - test - No observers have been added to this run
INFO - test - Running command 'main'
INFO - test - Started
INFO - test - Completed after 0:00:00

[thequilo]

I found the issue but I don't have a good solution for this yet. It is caused by the code around here: https://github.com/IDSIA/sacred/blob/89d8bb1d71b1732bf9735752524db71af9fe349c/sacred/config/utils.py#L109-L112. The return value of the config scope contains the full compiled config including the fixed (i.e., set via CLI) config values. When the second config scope is applied, it recieves these values in preset, so it thinks that the fixed values are set in the first config scope. It then shows up in the 'modified' section of the summary returned by the second config scope (and not in added as it should be) and error checking code doesn't recognize it as an added argument.