Initial implementation of sequential batch optimization

[nrontsis]

This is a first attempt to partially address #71

Minimal example (modified version of this):

import numpy as np
from GPflowOpt.domain import ContinuousParameter
import GPflow
from GPflowOpt.bo import BayesianOptimizer
from GPflowOpt.design import LatinHyperCube
from GPflowOpt.acquisition import QEI_CL


def fx(X):
    X = np.atleast_2d(X)
    # Return objective & gradient
    return np.sum(np.square(X), axis=1)[:, None]


domain = ContinuousParameter('x1', -2, 2) + ContinuousParameter('x2', -1, 2)

# Use standard Gaussian process Regression
lhd = LatinHyperCube(21, domain)
X = lhd.generate()
Y = fx(X)
model = GPflow.gpr.GPR(X, Y, GPflow.kernels.Matern52(2, ARD=True))

# Now create the Bayesian Optimizer
alpha = QEI_CL(model, batch_size=5)
optimizer = BayesianOptimizer(domain, alpha)

# Run the Bayesian optimization
# with optimizer.silent():
r = optimizer.optimize(fx, n_iter=15)
print(r)

In my setup, this PR and the original master code occasionally exhibits:

1. Optimisation failures

Warning: optimization restart #/# failed

2. NaN/Inf in the gradient

Warning: inf or nan in gradient: replacing with zeros

3. Cholesky decomposition failures.

[codecov-io]

Codecov Report

Merging #74 into master will decrease coverage by 1.72%. The diff coverage is 45.45%.

@@            Coverage Diff             @@
##           master      #74      +/-   ##
==========================================
- Coverage    99.8%   98.07%   -1.73%     
==========================================
  Files          17       18       +1     
  Lines        1013     1041      +28     
==========================================
+ Hits         1011     1021      +10     
- Misses          2       20      +18

Impacted Files	Coverage Δ
GPflowOpt/acquisition/__init__.py	100% <100%> (ø)	:arrow_up:
GPflowOpt/acquisition/acquisition.py	100% <100%> (ø)	:arrow_up:
GPflowOpt/acquisition/ei.py	100% <100%> (ø)	:arrow_up:
GPflowOpt/bo.py	89.28% <27.27%> (-9.39%)	:arrow_down:
GPflowOpt/acquisition/qei_cl.py	41.17% <41.17%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cc36cfe...40c79b8. Read the comment docs.

[javdrher]

Great job, this is great to start from. I think we can do some reshaping to make these features more part of the framework and less specific to the qEI implementation which is otherwise great. I'll add some notes inline.

For the troubles you encounter, 1-2 aren't something to be super worried about, 3 is concerning. This can usually be remedied by adding some priors on the hyperparameters (or some things we are discussing in #73 )