neural-cherche icon indicating copy to clipboard operation
neural-cherche copied to clipboard

Published 20 hours ago • verified publisher icon raphaelsty

Flops regularizer looks odd

Open dangkhoasdc opened this issue 8 months ago • 1 comments

This implement does not look similar to the formula mentioned in SPLADE paper. Also, to minimize this, the 2nd operand need to be equal to threshold, which is not the goal of FLOPS.

https://github.com/raphaelsty/neural-cherche/blob/2df02140ea7cda67ae348d690f17b1371e333827/neural_cherche/losses/flops.py#L94

There is another implementation more akin to the formula:

https://github.com/thongnt99/learned-sparse-retrieval/blob/d702026aacf1ab7c47011f55edcb2646a6bb646d/lsr/losses/regularizer.py#L56

dangkhoasdc avatar Mar 20 '25 03:03 dangkhoasdc

Hi @dangkhoasdc, I wrote this so it act as a margin-based flop loss, the model is asked to achieve a certain amount of flops.

Feel free to make a MR which will provide by default the correct flop loss :)

raphaelsty avatar Mar 20 '25 08:03 raphaelsty