huggingface
Automatically choose dataset split if none provided
Previously, evaluator.compute(..., data='imdb', ....) would fail because it was returning an object of type dataset.DatasetDict. This automatically detects a split if none is given (i.e. user passes in the dataset by name and expects the Evaluator to load it, instead of preloading it themselves).
Closes #226.
The documentation is not available anymore as the PR was closed or merged.
Hi @ola13, I actually put the original call to the parent class back in and returned the Dataset object instead, since I noticed that both the QA and token classification evaluators use columns from the dataset object in .compute() later. Let me know what you think!
Getting the strangest error in CI here: TestQuestionAnsweringEvaluator.test_model_init is failing in pytest with AssertionError: 33.333333333333336 != 0 but running the test suite in debug mode clearly shows the correct test results as expected, and self.data shows the correct two examples as expected. It almost feels like an environment/configuration error... digging deeper.
edit: resolved by #272
