Fixed: #1494 Check optimizer and learning rate settings

[jwliou]

Which issue(s) does this Pull Request fix?

Partial resolves #1494

Details of the Pull Request

In fact, this fix is more likely a resolving for #1480. (with implicit insertion of rmsprop and adadelta optimizers with default settings.)

This pull request is for checking if there already exist hyperparameter settings for optimizer and learning rate in current hp. Original code will just overwrite any possible optimizer and learning rate settings already in hp so that every model needs to search from a preset optimizer and learning rate space even if you set these in elsewhere such as tuners/task_specific.py.

It could be better to serialize optimizer settings, but where to put the serialization operations will be the next problem.

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Anselmoo]

See also: #1734

[jwliou]

Optimizer will be a larger block of settings including learning rate, instead of considering optimizer and learning rate separately.