adding resample benchmark objects
Description by Korbit AI
What change is being made?
Add resample benchmark objects ResampleBenchmark, AllTasksBenchmark, and HighVarianceBenchmark to the custom_benchmark.py file.
Why are these changes being made?
These changes introduce a new set of benchmark classes to support different evaluation strategies within the existing experimental framework. The ResampleBenchmark class provides a base implementation for resampling experiments, while AllTasksBenchmark and HighVarianceBenchmark implement specific evaluation and selection strategies for use cases that require comprehensive task inclusion and high variance filtering, respectively. This extensibility allows tailored analysis in experimental results handling.
Is this description stale? Ask me to generate a new description by commenting
/korbit-generate-pr-description