evaluate icon indicating copy to clipboard operation
evaluate copied to clipboard

Add missing metrics

Open sashavor opened this issue 2 years ago • 6 comments

As per @douwekiela's suggestion, we should find the blind spots that we have in terms of missing metrics, especially from domains like speech recognition and computer vision.

Suggestions are welcome below!

sashavor avatar Apr 19 '22 17:04 sashavor

We should probably look into GAN metrics as well, like Kernel Inception Distance (KID), Inception Score (IS) and Fréchet Inception Distance (FID) (maybe we should let people import them directly from a library like Torch Fidelity?

sashavor avatar Apr 19 '22 20:04 sashavor

How about RL metrics? e.g. https://analyticsindiamag.com/metrics-for-reinforcement-learning/

sashavor avatar Apr 19 '22 20:04 sashavor

Computer vision metrics: SSIM PSNR

There are various object detection metrics implemented by Tensorflow

sashavor avatar Apr 19 '22 20:04 sashavor

Also listed a few libraries in #11 e.g. NetworkX for graph metrics.

lvwerra avatar Apr 25 '22 08:04 lvwerra

For image generation, also consider LPIPS as well as unpaired metrics such as NIQE, PIQE, Brisque, SR-Metric

andreaferretti avatar Jun 13 '22 10:06 andreaferretti

This is also an interesting method/library for evaluating text generation https://github.com/neulab/BARTScore

cceyda avatar Jun 16 '22 16:06 cceyda