`X.random_evaluate()` broken with the `neldermead` example notebook.

[ChristopherMayes]

In scipy/neldermead.ipynb:

X.random_evaluate()

AssertionError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 X.random_evaluate()

File [~/Code/GitHub/Xopt/xopt/base.py:440](http://localhost:8890/~/Code/GitHub/Xopt/xopt/base.py#line=439), in Xopt.random_evaluate(self, n_samples, seed, custom_bounds)
    414 """
    415 Convenience method to generate random inputs using VOCs and evaluate them.
    416 
   (...)
    435 
    436 """
    437 random_inputs = self.vocs.random_inputs(
    438     n_samples, seed=seed, custom_bounds=custom_bounds, include_constants=True
    439 )
--> 440 result = self.evaluate_data(random_inputs)
    441 return result

File [~/Code/GitHub/Xopt/xopt/base.py:338](http://localhost:8890/~/Code/GitHub/Xopt/xopt/base.py#line=337), in Xopt.evaluate_data(self, input_data)
    335 # explode any list like results if all the output names exist
    336 output_data = explode_all_columns(output_data)
--> 338 self.add_data(output_data)
    340 # dump data to file if specified
    341 if self.dump_file is not None:

File [~/Code/GitHub/Xopt/xopt/base.py:369](http://localhost:8890/~/Code/GitHub/Xopt/xopt/base.py#line=368), in Xopt.add_data(self, new_data)
    367         new_data.index = new_data.index.astype(np.int64)
    368     self.data = new_data
--> 369 self.generator.add_data(new_data)

File [~/Code/GitHub/Xopt/xopt/generators/scipy/neldermead.py:137](http://localhost:8890/~/Code/GitHub/Xopt/xopt/generators/scipy/neldermead.py#line=136), in NelderMeadGenerator.add_data(self, new_data)
    134     return
    135 else:
    136     # Must have made at least 1 step, require future_state
--> 137     assert self.future_state is not None
    139     # new data -> advance state machine 1 step
    140     assert ndata == self.future_state.ngen == ngen + 1

AssertionError:

[ChristopherMayes]

@nikitakuklev it looks like this assertion comes from you.

[ChristopherMayes]

MWE:

from xopt import Xopt

YAML = """
generator:
  name: neldermead
  initial_point: {x0: -1, x1: -1}
evaluator:
  function: xopt.resources.test_functions.rosenbrock.evaluate_rosenbrock
vocs:
  variables:
    x0: [-5, 5]
    x1: [-5, 5]
  objectives: {y: MINIMIZE}
"""
X = Xopt(YAML)
X.random_evaluate()

[nikitakuklev]

Thanks, I'll try take a look this week. My instinct is that is the correct behavior - it makes no sense to do a random evaluation on simplex, since it must maintain state from one step to next. You can't add/remove/modify its data once started. Maybe a better error message is needed.

[roussel-ryan]

This is also breaking the current workflow from badger ATM. It would be great if this generator could treat the last datapoint in X.data as the starting point and then run from there (locking the dataframe in the process?)

[nikitakuklev]

The problem is that it is not just 'last point' but 'last simplex' + 'stage' (contraction, etc.). The implementation before current one did exactly this - it took last ndim+1 points and restarted with that as initial simplex. Because of that it was not reproducible on reload, whereas current implementation is (see here). This is fundamental difference from BO methods, which makes simplex a pain to handle with BO-like interface.

Maybe some override flags for old behavior are in order. I'll prototype a bit.

[roussel-ryan]

Thanks for looking into this Nikita, I think using multiple points may be helpful but are then causing the reproducibility issue you mentioned. If we just started simplex from a single point that would be ok as well if it fixes the reproducibility issues

[nikitakuklev]

Update: didn't have time implement a full solution yet,

[cr-xu]

Hi, I just ran into the same issue after updating to the new xopt version. Any plan to update/fix this?

Indeed, adding data points midway complicates the story (what is the current simplex etc..).

But, it would surely be reasonable to allow X.random_evaluate or X.evaluate({...}) for building up the initial simplex, so that we have a kind of consistent workflow for trying out different generators?

[roussel-ryan]

This has been a thorn in my side for quite awhile. I'll try taking another swing at this today.

[nikitakuklev]

Is the logic in #280 what you were looking for @cr-xu ?

For benchmarking, random initialization is a very bad idea because Nelder-Mead relies on having reasonably balanced/well-formed initial simplex. I think it should work/converge eventually no matter what. If you can instead generate a valid simplex (or take generator._initial_simplex after first step), and use that as initial points in other algos, should be more reasonable.

[cr-xu]

Yes, it looks pretty good. Thanks for updating this feature!