crowd-kit
crowd-kit copied to clipboard
[FEATURE] Unifying classification aggregators interfaces
Problem description
I wanted to test quality metrics of several different algorithms from crowdkit.aggregation.classification
and found myself writing such kind of function:
def get_scores(model, data, fit=True):
if fit:
model.fit(data)
probas = getattr(model, "probas_", None)
if probas is not None:
return probas
predictor = getattr(model, "predict_score", None)
if predictor is None:
predictor = model.predict_proba
return predictor(data)
That's because different models have different methods for retrieving scores. For example, MMSR
has predict_score
while almost all others have predict_proba
. Some have field probas_
, while others don't.
This seems strange and inconsistent.
Feature description
Unify naming of predict_score
functions and presence of probas_
field
Thanks for the issue! Will simply adding predict_proba
for M-MSR solve this? I assume we can just pass the scores through the softmax but I don't remember what these scores mathematically mean, need to read into the paper.
Thanks for the issue! Will simply adding
predict_proba
for M-MSR solve this? I assume we can just pass the scores through the softmax but I don't remember what these scores mathematically mean, need to read into the paper.
Yes, if all agreggators will have predict_proba
, the issue should be considered solved. Also, presence of probas_
field should be made consistent, as it was confusing for me why some models have it, and others don't
Also, KOS
aggregator does not have any probas at all. Is it algorithm's limitation or this can be added?
KOS does not operate with probabilities explicitly, but we can always throw NotImplementedError
if we find no reasonable workaround.