evaluate icon indicating copy to clipboard operation
evaluate copied to clipboard

Add `zero_division` argument to F1

Open l-uuz opened this issue 3 months ago • 0 comments

If you compute F1 in a case where a class is neither predicted nor present in the ground truth data, a warning refers to using zero_division:

import evaluate
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3]))

Output:

{'f1': array([0.57142857, 0.        , 0.        , 0.        ])}
[...\.venv\Lib\site-packages\sklearn\metrics\_classification.py:1731](file:///.../.venv/Lib/site-packages/sklearn/metrics/_classification.py:1731): UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, f"{metric.capitalize()} is", result.shape[0])

But this argument is not present:

import evaluate, numpy as np
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))

Output:

TypeError                                 Traceback (most recent call last)
Cell In[4], [line 3](vscode-notebook-cell:?execution_count=4&line=3)
      1 import evaluate, numpy as np
      2 f1 = evaluate.load("f1")
----> [3](vscode-notebook-cell:?execution_count=4&line=3) print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))

File ...\.venv\Lib\site-packages\evaluate\module.py:467, in EvaluationModule.compute(self, predictions, references, **kwargs)
    465 inputs = {input_name: self.data[input_name][:] for input_name in self._feature_names()}
    466 with temp_seed(self.seed):
--> [467](file:///.../.venv/Lib/site-packages/evaluate/module.py:467)     output = self._compute(**inputs, **compute_kwargs)
    469 if self.buf_writer is not None:
    470     self.buf_writer = None

TypeError: F1._compute() got an unexpected keyword argument 'zero_division'

l-uuz avatar Sep 07 '25 10:09 l-uuz