facebook
get_pareto_optimal_parameters(use_model_predictions=True) returns empty dictionary
Hello,
I am using the Service API for MOO with the default MOO model and I am unable to extract the Pareto optimal parameters with get_pareto_optimal_parameters(use_model_predictions=True). What could be the reason for that? The command returns me some parameterizations when I set use_model_predictions=False. However, they don't seem to align with the plot that I have created using the compute_posterior_pareto_frontier with that same model. Do you know why that might be?
Hi @IgorKuszczak, could you please post a fuller repro of what you are doing before calling get_pareto_optimal_parameters and what error you are encountering? It's possible that the model just hasn't been fit yet, that you are still in the Sobol phase of the optimization, or that your objective thresholds are configured too aggressively –– I will need to see more to be able to help.
The command returns me some parameterizations when I set use_model_predictions=False. However, they don't seem to align with the plot that I have created using the compute_posterior_pareto_frontier with that same model.
When use_model_predictions=False, we use raw metric values to produce the Pareto frontier, so it makes sense that it would differ from a model-produced one obtained with compute_posterior_pareto_frontier. With a fuller repro I'll be able to say more : )
Hello @lena-kashtelyan, thanks for responding so quickly! My implementation is very similar to the one shown in the Service API tutorials. The main difference is that I evaluate the objective function in external software and bring the results back to Python through a pickled dictionary. Additionally, I made the code to support parallel evaluations with sequential batch creation. I previously mentioned that exact implementation in #879 resulted in poor coverage of the Pareto front. I am interested in finding parametrization where both objective values are higher than 1, so I set the thresholds to 0.9. That being said, I also tried running the same optimization with a threshold set to 0 and had the same issues - poor coverage of the Pareto front and inability to extract the results with get_pareto_optimal_parameters. Maybe, the two issues are related. I tried to include all the relevant information in the code snippet below.
## Bayesian Optimization in Service API
NUM_SOBOL_STEPS = 10
NUM_OF_ITERS = 100
BATCH_SIZE = 1 # running sequential
# Generation strategy
gs = GenerationStrategy(steps=
[GenerationStep(model=Models.SOBOL, num_trials=NUM_SOBOL_STEPS),
GenerationStep(model=Models.MOO, num_trials=-1)])
# Initialize the ax client
ax_client = AxClient(generation_strategy=gs, random_seed=12345, verbose_logging=True)
# # Define the parameters
params = [
{
"name": "eta",
"type": "range",
"bounds": [0.1, 0.95],
"value_type": "float",
},
{
"name": "xi",
"type": "range",
"bounds": [0.1, 0.95],
"value_type": "float",
}
]
# # Creating the experiment
ax_client.create_experiment(
name="solid_hex_thick_opt",
parameters=params,
objectives={i: ObjectiveProperties(minimize=False, threshold=0.9) for i in ['stress_ratio', 'stiffness_ratio']},
outcome_constraints=[])
## Instantiate the Simulation object whose get_results method is used to evaluate the trials in an external software
sim = Simulation(...)
# The get_results method returns a dictionary {metric_name1: value1, metric_name2: value2}
# # Initialize the variables used in the iteration loop
abandoned_trials_count = 0
NUM_OF_BATCHES = NUM_OF_ITERS // BATCH_SIZE if NUM_OF_ITERS % BATCH_SIZE == 0 else NUM_OF_ITERS // BATCH_SIZE + 1
for i in range(NUM_OF_BATCHES):
try:
results = {}
trials_to_evaluate = {}
# Sequentially generate the batch
for j in range(min(NUM_OF_ITERS - i * BATCH_SIZE, BATCH_SIZE)):
parameterization, trial_index = ax_client.get_next_trial()
trials_to_evaluate[trial_index] = parameterization
# Evaluate the results in parallel and append results to a dictionary
for trial_index, parametrization in trials_to_evaluate.items():
with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:
try:
exec = executor.submit(sim.get_results, parametrization)
results.update({trial_index: exec.result()})
except Exception as e:
ax_client.abandon_trial(trial_index=trial_index)
abandoned_trials_count += 1
print(f'[WARNING] Abandoning trial {trial_index} due to processing errors.')
print(e)
if abandoned_trials_count > 0.1 * NUM_OF_ITERS:
print('[WARNING] More than 10 % of iterations were abandoned. Consider improving the parametrization.')
for trial_index in results:
ax_client.complete_trial(trial_index, results.get(trial_index))
except KeyboardInterrupt:
print('Program interrupted by user')
break
print(ax_client.get_pareto_optimal_parameters(use_model_predictions=True))
Hi @lena-kashtelyan, I feel like some of my problems might be a result of me misunderstanding what the get_pareto_optimal_parameters() function does. The documentation states that it identifies best parameterizations tried in the experiment so far. However, in lot of the trials that I do, the function returns an empty dictionary, even if I don't specify the thresholds (which rules out the theory that I am setting them to be too aggressive).
@IgorKuszczak, that definitely suggests a bug, and we will make sure to look into it! We are rather swamped right now, so it might take some time to investigate this, however.
Partial reproduction: It depends on the data
The issue must be data-dependent. I invented my own get_results function so I could run the code (runnable gist) and with the first function I tried, I didn't an empty Pareto frontier. @IgorKuszczak , if by any chance you have not moved on from this by now, it would be helpful if you could share the data generated by your trials by running print(ax_client.experiment.fetch_data().df).
But I did get an empty Pareto frontier for certain data, such as if outcomes were all 1's.
I'll look into this a bit more and possibly make a separate issue if appropriate.
Inferred thresholds
Just as an FYI, we can't really have no thresholds. If you set threshold = None, the thresholds will be inferred. This is because we can't calculate hypervolume without a threshold, and hypervolume improvement is used for acquiring new points. You can see if thresholds are being inferred with an INFO log like this, which I got when I set the data to be all 1s:
[INFO 09-26 15:12:14] ax.service.utils.best_point: Using inferred objective thresholds: [ObjectiveThreshold(stiffness_ratio >= 0.99999999), ObjectiveThreshold(stress_ratio >= 0.99999999)], as objective thresholds were not specified as part of the optimization configuration on the experiment.
The inferred thresholds ought to be chosen so that they're not removing data, and in the examples I looked at they were fine and this was not the culprit. More on the heuristic here.
The source of the bug is that the Pareto frontier is being computed on Y values in transformed space, but the thresholds remain in the original space (whether they are inferred or provided). Here's a repro:
def f():
NUM_SOBOL_STEPS = 5
NUM_OF_ITERS = 10
gs = GenerationStrategy(
steps=[
GenerationStep(model=Models.SOBOL, num_trials=NUM_SOBOL_STEPS),
GenerationStep(model=Models.MOO, num_trials=-1),
]
)
ax_client = AxClient(
generation_strategy=gs, random_seed=12345, verbose_logging=True
)
params = [
{
"name": name,
"type": "range",
"bounds": [0.1, 0.95],
"value_type": "float",
}
for name in ["eta", "xi"]
]
objective_names = ["stress_ratio", "stiffness_ratio"]
ax_client.create_experiment(
name="solid_hex_thick_opt",
parameters=params,
objectives={
i: ObjectiveProperties(minimize=False)
for i in objective_names
},
outcome_constraints=[],
)
# Y values range from 0 to 9
for i in range(NUM_OF_ITERS):
res = float(i)
results = {"stress_ratio": res, "stiffness_ratio": res}
_, trial_index = ax_client.get_next_trial()
ax_client.complete_trial(trial_index, results)
# Since data has been normalized, these values range from -1.46 to 1.46
print(ax_client.generation_strategy.model.model.Ys)
# returns an empty dictionary
# Info logs tell us that it's inferring thresholds of 7.596 for each metric
# Via pdb, I see that it's comparing the normalized Ys to the unnormalized threshold of 7.596
print(ax_client.get_pareto_optimal_parameters(use_model_predictions=True))
if __name__ == "__main__":
f()
In this example, the original Y values range from 0 to 9 for each metric. A threshold of 7.596 is inferred for each metric. But the Pareto frontier is being computed on normalized values, which range from -1.46 to 1.46. Since 1.46 is less than 7.596, none of the values qualifies for inclusion in the Pareto frontier.