Instance Weighting

[mossaab0]

🚀 Feature

Support providing weights for instances (not classes).

Motivation

As is the case with search engines, I have a dataset in which a document (e.g., query) might appear more than once. From a binary relevance stand point (i.e., the classification task), all occurrences of the same document have the same label. To save some memory and computation, instead of having a unique document d repeated N_d times in this dataset, and using those repeated instances to compute the relevant metrics (e.g., F_1 score), it's better to provide the array of weights as a parameter to use it as a multiplier for TP, TN, FP and FN.

Pitch

Add a parameter example_weight.

Alternatives

Repeat the documents N times. But this is not practical when some documents appear several orders of magnitudes compared to others (e.g., head vs tail queries in a search engine).

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[EvanZ]

I am using TweedieDevianceScore as a loss function and was also thinking along these lines. It would be really useful to be able to provide weights for each sample. Would the best way to do this currently to implement my own version of the TweedieDevianceScore as a custom metric?

[EvanZ]

It would be easy to incorporate weights if the metrics had an option to not reduce to a single scalar number, similar to setting reduction='none' in PyTorch loss functions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mossaab0]

Up (to avoid auto closing).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mossaab0]

Up (to avoid auto closing).

[bilelomrani1]

That would be a nice addition to the package. For reference, I believe this is related to #94