Simple example gets stuck

[Qwlouse]

For me this simple example tries two random values for f and then gets stuck always retrying the same value. Not sure if bug in Labwatch or in RoBO.

from sacred import Experiment
from labwatch import LabAssistant
from labwatch.hyperparameters import UniformInt
from labwatch.optimizers import BayesianOptimization

ex = Experiment()
la = LabAssistant(ex, database_name='labwatch_demo2',
                  optimizer=BayesianOptimization)


@ex.config
def cfg():
    f = 42


@la.searchspace
def small_search_space():
    f = UniformInt(lower=32, upper=64, default=32)


@ex.automain
def run(f):
    return f

Like this:

(labwatch) greff@Liz:~/Programming/labwatch/examples$ python my_example.py -p -d with small_search_space
WARNING: SMAC not found
Configuration (modified, added, typechanged, doc):
  f = 61
  seed = 862826954                   # the random seed for this experiment
-------------------------------------------------------------------------------
INFO - my_example - Running command 'run'
INFO - my_example - Started run with ID "1"
INFO - my_example - Result: 61
INFO - my_example - Completed after 0:00:00
(labwatch) greff@Liz:~/Programming/labwatch/examples$ python my_example.py -p -d with small_search_space
WARNING: SMAC not found
Configuration (modified, added, typechanged, doc):
  f = 52
  seed = 674124630                   # the random seed for this experiment
-------------------------------------------------------------------------------
INFO - my_example - Running command 'run'
INFO - my_example - Started run with ID "2"
INFO - my_example - Result: 52
INFO - my_example - Completed after 0:00:00
(labwatch) greff@Liz:~/Programming/labwatch/examples$ python my_example.py -p -d with small_search_space
WARNING: SMAC not found
Configuration (modified, added, typechanged, doc):
  f = 52
  seed = 886589901                   # the random seed for this experiment
-------------------------------------------------------------------------------
INFO - my_example - Running command 'run'
INFO - my_example - Started run with ID "3"
INFO - my_example - Result: 52
INFO - my_example - Completed after 0:00:00
(labwatch) greff@Liz:~/Programming/labwatch/examples$ python my_example.py -p -d with small_search_space
WARNING: SMAC not found
Configuration (modified, added, typechanged, doc):
  f = 52
  seed = 614908786                   # the random seed for this experiment
-------------------------------------------------------------------------------
INFO - my_example - Running command 'run'
INFO - my_example - Started run with ID "4"
INFO - my_example - Result: 52
INFO - my_example - Completed after 0:00:00

[Qwlouse]

The same problem seems to happen with the mlp example

[aaronkl]

Hmm for me the example works. I pushed my local changes could you pull it and try again?

[Qwlouse]

Seems better now, but still behaves different from what I'd expect:

{ "_id" : 1, "result" : 79 }
{ "_id" : 2, "result" : 83 }
{ "_id" : 3, "result" : 78 }
{ "_id" : 4, "result" : 31 }
{ "_id" : 5, "result" : 29 }
{ "_id" : 6, "result" : 1 }
{ "_id" : 7, "result" : 78 }
{ "_id" : 8, "result" : 79 }
{ "_id" : 9, "result" : 79 }
{ "_id" : 10, "result" : 78 }
{ "_id" : 11, "result" : 78 }
{ "_id" : 12, "result" : 78 }
{ "_id" : 13, "result" : 79 }
{ "_id" : 14, "result" : 78 }
{ "_id" : 15, "result" : 78 }
{ "_id" : 16, "result" : 30 }
{ "_id" : 17, "result" : 78 }
{ "_id" : 18, "result" : 78 }
{ "_id" : 19, "result" : 78 }
{ "_id" : 20, "result" : 78 }

[aaronkl]

It seems a bit odd that BO suggests configurations that are not in the bounds. Which version of RoBO do you use?

[Qwlouse]

Sorry, for that run I was using UniformInt(lower=1, upper=100). But it still seems strange to me that it would be so obsessed with 78.