Running Evaluation

[dhkhey]

Hi, Thanks for sharing this work. I am wondering if it is possible to run evaluation dataset to tune hyperparameters. The SetFitTrainer doesn't seem to accept arguments like 'evaluation_strategy', 'save_strategy', 'compute_metrics', etc. Or perhaps Im doing something wrong? Thanks.

[lewtun]

Hey @dhkhey, thanks for your interest!

The SetFitTrainer is indeed far less feature-complete than the one in transformers. In particular, we don't have an integration to tune hyperparameters with optuna or similar. Is there a type of evaluation that you're having trouble with?

Given how fast one can train SetFit models in the fewshot regime, we found it was quite quick to explore various hyperparameters . Having said that, if you would like to contribute a PR with hyperparameter search, we'd certainly welcome it!

[Mouhanedg56]

Hi @lewtun I'll be more than happy to add hyperparameter search in a PR.

[lewtun]

Amazing! I loosely modelled SetFitTrainer on transformers.Trainer, so having a similar API could make sense.

I also realised that we'll probably need a model_init arg in SetFitTrainer to be able to re-initialise the SetFitModel with each run

[Mouhanedg56]

Yeah, I noticed the huge difference between setfit.Trainer and transformers.Trainer. As you mentioned, we need to introduce model_init parameter in args before adding hyperparameter search. Can I lend a hand on this?

[lewtun]

@Mouhanedg56 yes, feel free to start with model_init :)

[Mouhanedg56]

Thanks @lewtun for approving #87 . Now with model_init supported, we can go forward and add a new PR for hyperparameter search.