verifiers
verifiers copied to clipboard
Disentangle scaffold and dataset
@daniel can add some color here as well, but the gist of it is not clear how to handle scaffolds/ datasets, e.g. for SWE environments we have
- scaffolds: mini swe agent plus, deepswe
- datasets: deepswe, swe bench verified, swe smith/ etc.
should there be one env per scaffold with configurable dataset? one env per dataset with configurable scaffold? one env per scaffold and dataset?
for reference take a look at deepswe or mini_swe_agent_plus environments.
or as discussed here