Q1: How do I remove a trial from a scheduler's experiment data?; Q2: How do I update the Generation strategy for a scheduler?

[mango-picket]

Hi, I wanted to ask for help for the following cases:

Q1: Let's assume that I ran a scheduler for 5 iterations. On the 5th iteration, due to errors in server, the job fails on the server. See image attached below:

If I resume my scheduler, I obtain an error in the get_job_result api (based on the scheduler tutorial, my server provides the get_job_result api). This happens because the server returns a None object due to failed job.

Is there a way for me to remove the last trial data or overwrite it with a (custom Trial) in the scheduler.experiment data? (assuming I don't want the trial to be abandoned, or failed). I am just wondering if its possible to edit the experiment data in the scheduler and if it's ok to do so?

Q2: Also, suppose, I would like to change the generation strategy that I passed initially to the scheduler, is it possible to update it? for eg. can I run a few iterations of SOBOL trials let's say at trial 16 from the above case?

Thanks for the help!

[bernardbeckerman]

Hi @mango-picket, thanks for asking this : )

assuming I don't want the trial to be abandoned, or failed

Why don't you want the trial to have a FAILED status? If you are ok with marking your trial as failed (in which case the scheduler can retry it as long as the optimizer re-suggests it) you can use experiment.trials[15].mark_failed(). Let me know if that works for your use case : )

[mango-picket]

Hi Bernard, Thanks for answering the question. Yes, the above modification helped as well as just adding the FAILED status. However, is there a reason to not modify experiment data directly -- as in removing the trial itself?

[bernardbeckerman]

@mango-picket the experiment is designed to be a record of all trials run, including failed/abandoned trials, so setting a status (e.g., FAILED) that causes Ax to ignore the trial is currently the best way to accomplish what you're trying to do. As things are currently set up, if you try to simply delete a trial, the experiment could become erroneous or get into an invalid state.

[deng-cy]

I also want to "delete" a trial. I am wondering about the difference between "FAILED" and "ABANDONED". Is there a doc describing how these statuses are treated by the generator? Does the generator ignore trials that are either "FAILED" or "ABONDONED"? If so, what is the difference between these two?

[bernardbeckerman]

The basic difference is that FAILED trials can be retried by the experiment, whereas ABANDONED trials will not be retried. See here for more details. Also, closing this out since it's been a couple months.