KeyError: 'Data point X is not unique'

[PoCk3T]

Hi and congrats for this sweet piece of software, really love it :)

I've implemented an async solution very similar to the one in the example notebook, with 13 virtual machines connecting to a master server hosting the Tornado webserver (like in the notebook), and it seems like I'm now constantly running over the same symptoms on the master server (the server registering the tested points and their respective target in the async example):

Error: 'Data point [3.47653914e+04 2.10539196e+02 3.15656650e+00 6.77134492e+00  1.01962491e+01] is not unique'
Traceback (most recent call last):
  File "playground_master.py", line 72, in post
    self._bo.register(params=body["params"], target=body["target"])
  File "/usr/local/lib/python3.5/dist-packages/bayes_opt/bayesian_optimization.py", line 104, in register
    self._space.register(params, target)
  File "/usr/local/lib/python3.5/dist-packages/bayes_opt/target_space.py", line 161, in register
    raise KeyError('Data point {} is not unique'.format(x))
KeyError: 'Data point [3.47653914e+04 2.10539196e+02 3.15656650e+00 6.77134492e+00 1.01962491e+01] is not unique'

It seems like the BO is suggesting to the 13 VMs to test points (set of 5 values in my case) which are already tested, almost in a constant manner after ~ 800 points tested.

Here's my troubleshooting so far:

• Using the load/save examples in the front page of BO, I save all points tested in a JSON file and for this instance of the problem, I see 817 points already registered in the JSON
• It seems like this data point thrown in the traceback is indeed ALREADY in the JSON, with an associated target value
• The 13 slave VMs are still trying to ask for "suggestions" (further set of points to test), but it seems like BO gives set of points already tested ; I still see some rare cases when it's not tested yet and the counts case increase slightly to 818, 819 points.. (but most of the time the traceback is thrown)

I'm a little bit surprised you can end up with such scenario given my PBOUND is very broad, and so, has a lot to points to be worked on without having to test the same ones again:

            {'learning_timesteps': (5000, 40000),
            'timesteps_per_batch': (4, 72),
            'observation_loopback': (1, 20),
            'mmr__timeperiod': (7, 365),
            'model': (-0.49, 5.49) }

This is how I initialized the utility, which, as far as I understood, is responsible for suggesting points that were already tested: _uf = UtilityFunction(kind="ucb", kappa=2.576, xi=0)

Should I modify the acquisition function? or some of the hyperparameters "kappa", "xi" ?

I see https://github.com/fmfn/BayesianOptimization/issues/92 related to this but I'm not doing any manual point probing / not doing any initialization, I really sticked to the Async notebook example, so I'm not sure this issue applies to me :(

Let me know if I can share further information & more context to this Thanks in advance for the help :) Lucas

[fmfn]

Yeah, that's an annoying, somewhat obscure bug, that comes back every now and then. I haven't been able to reproduce it consistently, so it's hard to track it down. If you could come up with a script that always leads to this bug, please share (either here or shoot me an email).

I have a few suspicions as to why it happens, but haven't confirm them. I also have in mind a couple of ways to patching it, in the sense of going over the bug not necessarily fixing it (since I don't really know why/when it happens). So that's an option.

Are you catching these KeyErrors and moving on with the process? I'm interested in knowing if you reach a point where no more new suggestions are being given and the whole comes to a halt.

[PoCk3T]

Thanks @fmfn .

Yeah it almost comes to an halt, because the "slaves" asks for points to test, work on them, send back target result, but the KeyError is throwed (and caught, so the server can keep serving other requests). Thus the optimization as such is on a halt.

Sure I can try to make a minimal version to reproduce this if this keeps happening ; overnight, I've tested "ei" instead of "ucb", but same issues happened, and now I'm back to "ucb" with a XI value of 0.01 instead of 0.00 and watching. As I was reading other issues around here, I also went ahead and changed -0.49 to 0.0 in one my boundaries in order to avoid any potential issues related to negative boundaries. Will keep you posted

Would it help if I share a JSON where problem happens ? My boundaries and indicator function are already indicated in this thread, so with that three elements that should be the easiest to reproduce, or am I missing something ? :)

Thanks again!

[fmfn]

The xi parameters is not used by UCB, so I don't expect that to make a different. If you change kappa you may be able to "fix" it.

The JSON should be enough for now. I want to probe if the problem is with maximizing the acquisition function. If that's the case, it could just be a matter of increasing kappa to get the ball rolling again, maybe you can start trying that.

[PoCk3T]

Small update of the day:

• Used 3.0 for Kappa on UCB, issue still appeared and I couldn't get over 809 points tested for a whole day, it was stuck on those
• I've uploaded the content of the JSON saving all steps and points here: https://jpst.it/1HyaZ

Please let me know if I should try with another Kappa value or if a minimal way to reproduce it is needed (it's basically this JSON + the boundaries and I indicated in my original post)

[wg12385]

Hi, i've stumbled on this bug myself (big fan of the software otherwise !). Have either of you found a work-around yet ?

If it helps i've tried handling the KeyError and increasing the kappa value then trying again and it doesn't make any difference for me, once I get one identical point, it will ONLY produce identical points from then on.

Also this might be relevant, for me I can tell that it is going to produce the error before it tries, because the suggested parameters are all "round" numbers, (1.e-1 rather than 1.39475739e-1, etc), though i've got no idea what that could mean.

[yfszzx]

I met same problem, and after hours debug, I guess that when we load old points, every time method “register”is used, method " suggest" must be used once, though we need no new point in the process. Now it works well. I think when “suggest” is used, some importent inner calculation is running and the inner result is recorded, which is importent for next "suggest"

[bwheelz36]

Hey all, I am also running into this problem. The below script consistently reproduces the error for me. As a workaround, I can replace the error statement in line 163 of target_space with a warning statement. But that still results in the same point being assessed again and again... Another easy alternative would be to augment the repeated point with some random noise? Some other thoughts:

• it appears this behavior arises when the problem is basically solved, i.e. there are no other points the algorithm thinks are interesting. I'm not sure if that's helpful or not....
• Could this be associated with the 'edge obsession' that is reported in many issues here?

import numpy as np
from bayes_opt import BayesianOptimization
from bayes_opt import UtilityFunction


def f(x):
    return np.exp(-(x - 2) ** 2) + np.exp(-(x - 6) ** 2 / 10) + 1/ (x ** 2 + 1)


if __name__ == '__main__':
    optimizer = BayesianOptimization(f=None, pbounds={'x': (-2, 2)}, verbose=2, random_state=1)
    optimizer.set_gp_params(normalize_y=True, alpha=2.5e-3, n_restarts_optimizer=20)  # tuning of the gaussian parameters...
    utility = UtilityFunction(kind="ucb", kappa=5, xi=1)  # kappa determines explore/Exploitation ratio
    for point in range(20):
        next_point_to_probe = optimizer.suggest(utility)
        NextPointValues = np.array(list(next_point_to_probe.values()))
        mean,std = optimizer._gp.predict(NextPointValues.reshape(1, -1),return_std=True)
        target = f(**next_point_to_probe)
        optimizer.register(params=next_point_to_probe, target=target)

[zwelz3]

I saw this issue and I think I have an example that consistently produces the error. Running on Windows 10, version 1.1.0 of this package.

I may be setting alpha too high. Lowering alpha does not produce this issue.

Please ignore the gross research nature of the code. I was playing around with tweaking tunable parameters in a notebook.

import numpy as np


def black_box_function(x, epsilon=0.1):
    """Function with unknown internals we wish to minimize.
    
    :param epsilon: the value of the isotropic noise for the objective function

    This is just serving as an example, for all intents and
    purposes think of the internals of this function, i.e.: the process
    which generates its output values, as unknown.
    """
    response = (np.exp(-(x - 2)**2) + np.exp(-(x - 6)**2/4) + 1/ (x**2 + 1))  # change 2/4 -> 2/10 for easier objective
    try:
        noise_vec = [np.random.randn()*epsilon for _ in range(len(x))]
        return [sum(vals) for vals in zip(response, noise_vec)]
    except TypeError:
        return response + np.random.randn()*epsilon

from bayes_opt import BayesianOptimization, UtilityFunction

# Number of optimization iterations
n_iter = 100

# Bounded region of parameter space
pbounds = {'x': (-2, 10)}

x = np.linspace(pbounds['x'][0], pbounds['x'][1], 100).reshape(-1, 1)
y = black_box_function(x, epsilon=0)  # the actual function value for each x

import matplotlib.pyplot as plt
from matplotlib import gridspec

plt.plot(x, y)
plt.plot(x, black_box_function(x))
plt.legend(["f(x)", "f(x) w/ noise"])


from bayes_opt import UtilityFunction


def posterior(optimizer, x_obs, y_obs, grid):
    optimizer._gp.fit(x_obs, y_obs)

    mu, sigma = optimizer._gp.predict(grid, return_std=True)
    return mu, sigma


def plot_gp(optimizer, utility_function, x, y):
    fig = plt.figure(figsize=(16, 10))
    steps = len(optimizer.space)
    fig.suptitle(
        'Gaussian Process and Utility Function After {} Steps'.format(steps),
        fontdict={'size':30}
    )
    
    gs = gridspec.GridSpec(2, 1, height_ratios=[3, 1]) 
    axis = plt.subplot(gs[0])
    acq = plt.subplot(gs[1])
    
    x_obs = np.array([[res["params"]["x"]] for res in optimizer.res])
    y_obs = np.array([res["target"] for res in optimizer.res])
    
    mu, sigma = posterior(optimizer, x_obs, y_obs, x)
    axis.plot(x, y, linewidth=3, label='Target')
    axis.plot(x_obs.flatten(), y_obs, 'D', markersize=8, label=u'Observations', color='r')
    axis.plot(x, mu, '--', color='k', label='Prediction')

    axis.fill(np.concatenate([x, x[::-1]]), 
              np.concatenate([mu - 1.9600 * sigma, (mu + 1.9600 * sigma)[::-1]]),
        alpha=.6, fc='c', ec='None', label='95% confidence interval')
    
    axis.set_xlim((-2, 10))
    axis.set_ylim((None, None))
    axis.set_ylabel('f(x)', fontdict={'size':20})
    axis.set_xlabel('x', fontdict={'size':20})
    
    utility = utility_function.utility(x, optimizer._gp, 0)
    acq.plot(x, utility, label='Utility Function', color='purple')
    acq.plot(x[np.argmax(utility)], np.max(utility), '*', markersize=15, 
             label=u'Next Best Guess', markerfacecolor='gold', markeredgecolor='k', markeredgewidth=1)
    acq.set_xlim((-2, 10))
    acq.set_ylim((0, np.max(utility) + 0.5))
    acq.set_ylabel('Utility', fontdict={'size':20})
    acq.set_xlabel('x', fontdict={'size':20})
    
    axis.legend(loc=2, bbox_to_anchor=(1.01, 1), borderaxespad=0.)
    acq.legend(loc=2, bbox_to_anchor=(1.01, 1), borderaxespad=0.)

import ipywidgets as ipyw

progbar = ipyw.IntProgress(max=n_iter-1, val=0)

random_state = 3

# specify tuning parameters
alpha = 1
length_scale = 0.1
kappa = 15

import numpy as np
from sklearn.gaussian_process.kernels import ConstantKernel, RBF


optimizer = BayesianOptimization(
    f=None,  # We don't want it to control the objective function internally
    pbounds=pbounds,
    random_state=3,
    verbose=0  # No printouts at each step of the optimization
)
# Length scale should be modified if the response surface is "spiky" vs "smooth"
optimizer.kernel = rbf = ConstantKernel(1.0) * RBF(length_scale=length_scale)
# use default if there is no noise in the objective function, else set this to something larger
optimizer._gp.alpha = alpha

# Instantiate the utility function (so we can manually suggest the next point to probe)
utility = UtilityFunction(kind="ucb", kappa=kappa, xi=0.0)

print("Optimization Progress:")
progbar

# Custom maximize loop (suggest, probe, register)
# optimizer.maximize(init_points, n_iter, kappa)
pre_alpha = np.zeros(n_iter)  # pre_allocated (for speed)
n_init = max(2, round(n_iter/100))  # the number of points to sample before updating the prior/utility
progbar.value = 0

# Get initial sample points (should be random)
init_samples = []
for idx in range(n_init):
    init_samples.append(optimizer.suggest(utility))
    
# Run through initial samples
for next_point in init_samples:
    response = black_box_function(**next_point)
    obj_mu = response  # TODO remove
    # Tell the optimizer what the results of the objective function evaluation were
    optimizer.register(params=next_point, target=obj_mu)
    
    progbar.value += 1
    

# Run through remaining samples
# TODO could implement a kappa decay here (e.g. 0.9 or something to shift towards exploitation in later samples)
for idx in range(n_init, n_iter):
    # Get next point to sample from the utility function
    next_point = optimizer.suggest(utility)
    
    # Compute the target (in batch this returns a mu, sigma TODO)
    response = black_box_function(**next_point)
    # TODO obj_mu, obj_sigma = response
    obj_mu = response  # TODO remove
    #pre_alpha[idx] = obj_sigma = 0.1  # TODO remove 0.1
    
    # Tell the optimizer what the results of the objective function evaluation were
    optimizer.register(params=next_point, target=obj_mu)
    
    #optimizer._gp.alpha = pre_alpha[:idx+1]
    
    progbar.value += 1

plot_gp(optimizer, utility, x, y)

I've added a zip of the notebook which is probably easier to run than the paste above. NoisyObj_ManualLoop.zip

[rmcconke]

I have also run into this issue, but with a much simpler example. It seems like this issue occurs when the same point is registered twice.

from bayes_opt import BayesianOptimization
from bayes_opt import UtilityFunction
import numpy as np

def black_box_function_sim_dummy(coef):
    print(coef)
    result = sum(coef.values())
    return result #+np.random.uniform(0.01,0.1)

coef = {
    'a1': (0.2,0.5),
}

optimizer = BayesianOptimization(
    f=None,
    pbounds=coef,
    verbose=2,
    random_state=1,
)

utility = UtilityFunction(kind="ucb", kappa=2.5, xi=0.0)

for i in range(10):
    next_point = optimizer.suggest(utility)
    print(next_point)
    target = black_box_function_sim_dummy(next_point)
    optimizer.register(params=next_point, target=target)
    
    print(target, next_point)
    
print(optimizer.max)

When I leave the random noise out of the black box function, I get this error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [22], in <cell line: 23>()
     25     print(next_point)
     26     target = black_box_function_sim_dummy(next_point)
---> 27     optimizer.register(params=next_point, target=target)
     29     print(target, next_point)
     31 print(optimizer.max)

File ~/WDK/ML/rans_tuner/tuner/lib/python3.9/site-packages/bayes_opt/bayesian_optimization.py:108, in BayesianOptimization.register(self, params, target)
    106 def register(self, params, target):
    107     """Expect observation with known target"""
--> 108     self._space.register(params, target)
    109     self.dispatch(Events.OPTIMIZATION_STEP)

File ~/WDK/ML/rans_tuner/tuner/lib/python3.9/site-packages/bayes_opt/target_space.py:161, in TargetSpace.register(self, params, target)
    159 x = self._as_array(params)
    160 if x in self:
--> 161     raise KeyError('Data point {} is not unique'.format(x))
    163 # Insert data into unique dictionary
    164 self._cache[_hashable(x.ravel())] = target

KeyError: 'Data point [0.5] is not unique'

When I uncomment that random noise in the result, it runs fine. So, it seems this issue occurs when the optimizer converges to a similar value on the bounds for my case.

[bwheelz36]

Indeed. As I note above, this seems to occur when the problem is already solved - the optimizer sees no more value in probing points other than the ones it has already probed. To solve this problem in my application, I just put a try/catch statement which I've copied and pasted below which will allow a few 'same points' to be probed, before quitting if the behavior keeps occurring. In my work, when the same point starts to get repeatedly probed it is a good indication of convergence so I am happy to just stop the process at that point.

I think the question is how we should actually handle this - e.g. we could attempt to put something like the code block below into the main code such that it will throw some warnings a few times and then terminate? It's not totally clear to me what the 'correct' behavior should be...

try:
    self.optimizer.register(params=next_point_to_probe, target=target)
except KeyError:
    try:
        self.RepeatedPointsProbed = self.RepeatedPointsProbed + 1
        logger.warning(
            f'Bayesian algorithm is attempting to probe an existing point: {NextPointValues}. Continuing for now....')
        if self.RepeatedPointsProbed > 10:
            logger.error('The same point has been requested more than 10 times; quitting')
            break
    except AttributeError:
        self.RepeatedPointsProbed = 1

[rmcconke]

It seems this could be part of a larger feature related to a "termination" condition - AFAIK, the current code only runs for a specified number of iterations, it does not have a "convergence criteria". The error arises from the register function finding an identical point, so when the optimizer gets "stuck" here I think the same point will be repeatedly suggested.

After suggesting a duplicate point, the point is not registered (the try statement fails), and it will suggest the same point on the next iteration (the posterior hasn't changed). So, it may be possible to detect immediately (i.e., after only a single duplicate suggestion). It isn't really an error, it is the optimizer getting stuck at a point where the utility function is higher than any other unprobed location.

This tends to happen with highly exploitative values of kappa or xi, so we could also suggest a higher value in the error message if this occurs.

[bwheelz36]

fixed in #372

[2258324319]

Thanks for bwhelz36's work, this problem has been solved on bayesian-optimazition>=1.4.0 if your python>=3.7, pip install bayesian-optimazition==1.4.2 if you are using python36 or python35 like me, this doesn't work. Because they add colorama 0.4.6, which is not suitable for us. but you can install bwheelz's work in which he solved this bug first: pip install bayesian-optimazition==1.4.0