IDSIA
Call captured functions from code & nicer interface to mongodb
Hello developers of sacred! First, I want to thank you for this awesome project. It is so helpful for keeping track of your experiments.
I currently run my experiments from the commandline and want to access the results using an ipython notebook.
However, it is not clear to me how to load an experiment and how to call captured functions. Here, my current approach and why it does not work.
My experiment file:
ex = Experiment()
@ex.capture
def create_model(model_hyperparams, datset):
// build model
return model
....
In the Jupyter Notebook, I would like to create the model from the save config:
from my_experiment import create_model
client = MongoClient()
db = client.get_database('sacred')
exps = db.runs.find({
'experiment.name': {'$eq': 'train_area_models_4x4'},
'config.use_conditional': False,
'status': 'COMPLETED',
})
ex = next(exps)
create_model(
model_hyperparams=ex['config']['model_hyperparams'],
dataset=ex['config']['dataset'],
...
)
Then I get this error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-40-e5e56d899fb3> in <module>()
3 dataset=ex['config']['dataset'],
4 use_conditional=ex['config']['use_conditional'],
----> 5 encoder=ex['config']['encoder'],
6 )
~/.local/lib/python3.5/site-packages/wrapt/wrappers.py in __call__(self, *args, **kwargs)
521
522 return self._self_wrapper(self.__wrapped__, self._self_instance,
--> 523 args, kwargs)
524
525 class BoundFunctionWrapper(_FunctionWrapperBase):
~/.local/lib/python3.5/site-packages/sacred/config/captured_function.py in captured_function(wrapped, instance, args, kwargs)
42 args, kwargs = wrapped.signature.construct_arguments(args, kwargs, options,
43 bound)
---> 44 wrapped.logger.debug("Started")
45 start_time = time.time()
46 # =================== run actual function =================================
AttributeError: 'NoneType' object has no attribute 'debug'
It seems to me that the experiment must be running to call captured functions.
I guess a simple fix to this problem would be to change sacred/config/captured_function.py:
@wrapt.decorator
def captured_function(wrapped, instance, args, kwargs):
options = FallbackDict(
wrapped.config,
_config=wrapped.config,
_log=wrapped.logger,
_run=wrapped.run
)
if wrapped.uses_randomness: # only generate _seed and _rnd if needed
options['_seed'] = get_seed(wrapped.rnd)
options['_rnd'] = create_rnd(options['_seed'])
bound = (instance is not None)
args, kwargs = wrapped.signature.construct_arguments(args, kwargs, options,
bound)
if wrapped.logger is not None:
wrapped.logger.debug("Started")
start_time = time.time()
# =================== run actual function =================================
result = wrapped(*args, **kwargs)
# =========================================================================
if wrapped.logger is not None:
stop_time = time.time()
elapsed_time = timedelta(seconds=round(stop_time - start_time))
wrapped.logger.debug("Finished after %s.", elapsed_time)
return result
A even nicer thing would be to have a sacred interface to MongoDB like this:
// wrapper of mongodb
db = SacredMongoDB()
db.find({
'experiment.name': {'$eq': 'train_area_models_4x4'},
'config.use_conditional': False,
})
# get one experiment
ex = next(db)
# access config
ex.config.use_conditional # -> False
# access experiment values
ex.experiment.name # -> train_area_models_4x4
# call captured functions with the config from the database
ex.create_model()
# raises Exception. Experiment already run
ex.run()
I probably could create such a wrapper for mongodb if you think this is useful to have for sacred.
My workflow for doing exactly this is the following:
from my_experiment import ex, create_model
from pymongo import MongoClient
client = MongoClient()
run_entry = client.sacred.find_one({
'experiment.name': 'train_area_models_4x4',
'config.use_conditional': False,
'status': 'COMPLETED',
})
# create a run object
run = ex._create_run(config_updates=run_entry['config'])
# Access config and lots of other things:
run.config['use_conditional'] # -> False
# now you can call captured functions that will use that configuration
create_model()
# or execute the main function:
run()
I agree that having a better interface to instantiate runs from the database would be nice. But this goes hand in hand with providing a more object-oriented interface for sacred (see #193 ) and involves quite a bit of refactoring. It is on my long-term roadmap, but will probably take a while.
