AgentLab icon indicating copy to clipboard operation
AgentLab copied to clipboard

adding resample benchmark objects

Open TLSDC opened this issue 9 months ago • 0 comments

Description by Korbit AI

What change is being made?

Add resample benchmark objects ResampleBenchmark, AllTasksBenchmark, and HighVarianceBenchmark to the custom_benchmark.py file.

Why are these changes being made?

These changes introduce a new set of benchmark classes to support different evaluation strategies within the existing experimental framework. The ResampleBenchmark class provides a base implementation for resampling experiments, while AllTasksBenchmark and HighVarianceBenchmark implement specific evaluation and selection strategies for use cases that require comprehensive task inclusion and high variance filtering, respectively. This extensibility allows tailored analysis in experimental results handling.

Is this description stale? Ask me to generate a new description by commenting /korbit-generate-pr-description

TLSDC avatar Feb 26 '25 16:02 TLSDC