Ax
Ax copied to clipboard
facebook
Get rid of SurrogateRunner and SurrogateBenchmarkProblem
Summary:
Context: This diff cuts over over uses of SurrogateRunner to use ParamBasedTestProblemRunner with a test_problem that is the newly introduced SurrogateTestFunction, and the following diff after that will bring us down to only one runner class for benchmarking by merging ParamBasedTestProblemRunner into BenchmarkRunner. Having only one runner will make it easier to enable asynchronous benchmarks. Currently, SurrogateRunner had its own logic for tracking when trials are completed, which would make it difficult to work in with asynchronicity.
Note on naming: Some names have become non-intuitive in the process of benchmarking. To accord with some future changes I hope to make, I called a new class SurrogateTestFunction, whereas SurrogateParamBasedTestProblem would be more in line with the old naming.
The name changes I hope to make:
- ParamBasedTestProblemRunner -> nothing, absorbed into BenchmarkRunner
- ParamBasedTestProblem -> TestFunction, to emphasize that all it does is generate data (rather than more generally specify the problem we are solving) and that it is deterministic, and to differentiate it from BenchmarkProblem. BenchmarkTestFunction would also be a candidate.
- BoTorchTestProblem -> BoTorchTestFunction
Changes in this diff:
- Introduces SurrogateTestFunction, a ParamBasedTestProblem for * Removes SurrogateRunner. There is some loss of (unused) functionality here; dict-valued 'noise_std' is no longer supported. I expect we will eventually want that, but it doesn't need to be done now.
- Changed
predict_for_tensorutility function to use aSurrogateTestFunctionrather than aSurrogateRunner - Removed
SurrogateRunnerfrom the json_store registry. There is no point in keeping it around for backward compatibility since it was just deserializing to a dummySyntheticRunneranyway.
Differential Revision: D64855010
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 95.66%. Comparing base (
d1e65c5) to head (712a0e1). Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #2954 +/- ##
========================================
Coverage 95.66% 95.66%
========================================
Files 486 485 -1
Lines 48944 48832 -112
========================================
- Hits 46821 46716 -105
+ Misses 2123 2116 -7
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
