BayesianOptimization Tuner doesn't return the model with the best val_loss and "Best val_loss So Far" increases sometimes

[douira]

The BayesianOptimization tuner doesn't return the model with the best val_loss and "Best val_loss So Far" increases sometimes. Am I misunderstanding how the tuner works? Wouldn't the "Best val_loss So Far" never increase again?

Here's an excerpt of the log: (I've omitted log lines from training with ...)

Search: Running Trial #62

Hyperparameter    |Value             |Best Value So Far 
lstm_reg          |0.01              |0                 
lstm_units        |384               |416               
learning_rate     |0.01741           |0.00062759        

Epoch 1/200
58/58 - 8s - loss: 5.8378 - mean_absolute_error: 0.8131 - val_loss: 2.1253 - val_mean_absolute_error: 0.6561
...
Epoch 26/200
58/58 - 5s - loss: 0.4074 - mean_absolute_error: 0.4579 - val_loss: 0.8352 - val_mean_absolute_error: 0.5948
Trial 62 Complete [00h 02m 37s]
val_loss: 0.5230200886726379

Best val_loss So Far: 0.396116703748703
Total elapsed time: 04h 32m 29s

Search: Running Trial #63

Hyperparameter    |Value             |Best Value So Far 
lstm_reg          |0.001             |0                 
lstm_units        |288               |416               
learning_rate     |0.00073415        |0.00062759        

Epoch 1/200
58/58 - 5s - loss: 0.8142 - mean_absolute_error: 0.6041 - val_loss: 0.8935 - val_mean_absolute_error: 0.5796
...
Epoch 45/200
58/58 - 5s - loss: 0.1761 - mean_absolute_error: 0.2561 - val_loss: 0.8256 - val_mean_absolute_error: 0.6804
Trial 63 Complete [00h 04m 04s]
val_loss: 0.527589738368988

Best val_loss So Far: 0.396116703748703
Total elapsed time: 04h 36m 34s

Search: Running Trial #64

Hyperparameter    |Value             |Best Value So Far 
lstm_reg          |0.01              |0                 
lstm_units        |384               |416               
learning_rate     |0.00011261        |0.00062759        

Epoch 1/200
58/58 - 6s - loss: 4.1151 - mean_absolute_error: 0.6866 - val_loss: 3.3185 - val_mean_absolute_error: 0.4880
...
Epoch 94/200
58/58 - 6s - loss: 0.3712 - mean_absolute_error: 0.3964 - val_loss: 0.7933 - val_mean_absolute_error: 0.5781
Trial 64 Complete [00h 09m 06s]
val_loss: 0.6574578285217285

Best val_loss So Far: 0.43126755952835083
Total elapsed time: 04h 45m 40s

Search: Running Trial #65

Hyperparameter    |Value             |Best Value So Far 
lstm_reg          |0.0001            |0                 
lstm_units        |480               |256               
learning_rate     |0.010597          |0.05              

Epoch 1/200
58/58 - 6s - loss: 1.1511 - mean_absolute_error: 0.7090 - val_loss: 1.1972 - val_mean_absolute_error: 0.6724
...

[haifeng-jin]

Thanks for the issue! Is there a Colab notebook for reproducing this bug with a fixed seed?

[douira]

No not right now because this occurred in a project. I can look into making a reproduction notebook but it might take time.

[douira]

I'm working on creating a notebook for reproducing this issue but it's not easy. In the mean time, do you think the fact that I'm using executions_per_trial=3 could have something to do with this? Does the printed val_loss after "trial complete" actually reflect the average loss of the three trained models or is it just the loss of the last model? If that is the case, then the tuner itself might be working correctly but the printing would be wrong.

[haifeng-jin]

@douira After inspecting the code, this is exactly what is happening. It is using the average of multiple executions as the objective value.

We should fix the printing. The best value should never increase.

[douira]

Thanks for finding this issue in the code! Now I don’t have to worry that something is being calculated wrong. Fixing the printing is a good idea!