Problems integrating with BOHB

[PatrykChrabaszcz]

I noticed that there is a problem with ordinal hyperparameters. I do not know if it is a bug or a feature but they do not behave like integer hyperparameters.

Here is an example of what happens: Define ConfigSpace in the .pcs file:

batch_size integer [8, 128] [64] log sequence_size ordinal {128, 256, 512, 1024, 2048, 4096} [512]

Load this ConfigSpace and then sample random values and convert to a vector: config_space = ... # Some loading code sample_vector = config_space.sample_configuration().get_array()

#Corresponding Array: #[ 0.40941199 2. ]

Now convert back to the Configuration: sample = Configuration(config_space, vector=sample_vector)

Works nicely: batch_size, Value: 24 sequence_size, Value: '512'*

Add some noise to the vector (BOHB does that when sampling from the model, it treats ordinal, integer and continuous parameters in the same way) sample_vector[0] += 0.1 sample_vector[1] += 0.1

#Corresponding Array: #[ 0.50941199 2.1 ]

Now convert back to the Configuration: sample = Configuration(config_space, vector=sample_vector)

For some reason sequence_size disappears. Configuration: batch_size, Value: 32

Is that intended? Shouldn't ordinal hyperparameter have the same representation as integer hyperparameter when transformed to an array? Now it has representation that is similar to categorical hyperparameter.

[mfeurer]

Thanks a lot for reporting this. I can reproduce the issue on my computer:

from ConfigSpace import (
    OrdinalHyperparameter,
    UniformIntegerHyperparameter,
    ConfigurationSpace,
    Configuration,
)

hp1 = UniformIntegerHyperparameter('batch_size', lower=8, upper=128, log=True)
hp2 = OrdinalHyperparameter('sequence_size', (128, 256, 512, 1024, 2048, 4096))
cs = ConfigurationSpace()
cs.add_hyperparameters([hp1, hp2])
sample_vector = cs.sample_configuration().get_array()
sample = Configuration(cs, vector=sample_vector)
print(sample)
sample_vector[0] += 0.1
sample_vector[1] += 0.1
sample = Configuration(cs, vector=sample_vector)
print(sample)

results in

Configuration:
  batch_size, Value: 76
  sequence_size, Value: 2048

Configuration:
  batch_size, Value: 101

I think there are multiple underlying issues:

• We don't document that there are no input checks conducted when passing a vector, only when passing raw values
• The ordinal hyperparameter doesn't complain when one tries access an illegal value.

[eddiebergman]

Not sure when but in #346, doing so raises the foloowing error:

Configuration(values={
  'batch_size': 18,
  'sequence_size': 128,
})
Traceback (most recent call last):
  File "/home/skantify/code/ConfigSpace/h.py", line 20, in <module>
    print(sample)
  File "/home/skantify/code/ConfigSpace/src/ConfigSpace/configuration.py", line 236, in __str__
    values = dict(self)
  File "/home/skantify/code/ConfigSpace/src/ConfigSpace/configuration.py", line 215, in __getitem__
    value = hyperparameter.to_value(vector)
  File "/home/skantify/code/ConfigSpace/src/ConfigSpace/hyperparameters/hyperparameter.py", line 235, in to_value
    value = self._transformer.to_value(np.array([vector]))[0]
  File "/home/skantify/code/ConfigSpace/src/ConfigSpace/hyperparameters/_hp_components.py", line 86, in to_value
    raise ValueError(
ValueError: Got unexpected float value while trying to transform a vector representation into a value in [ 128  256  512 1024 2048 4096].Expected integers but got [0.1] (dtype: float64)

However I believe that we should add a boolean flag when creating a Configuration from a vector that will trigger validation. We can set this flag internally where appropriate, so as not to lose speed in critical operations. Will add it to the todo of #346