open_clip icon indicating copy to clipboard operation
open_clip copied to clipboard

fix repetition 'sunglasses' label

Open darkasevgen opened this issue 11 months ago • 1 comments

There was a repetition "sunglasses" at IMAGENET_CLASSES. As far as I understand, these class names have already been corrected (for instance, 'crane' transform into "crane bird" and "construction crane", but "sunglasses" haven't been). Reference: https://www.kaggle.com/c/imagenet-object-localization-challenge/data?select=LOC_synset_mapping.txt

The desired mapping {folder name: class name}: n04355933 sunglass n04356056 sunglasses, dark glasses, shades

I have added this difference so that there are 1000 unique labels.

darkasevgen avatar Mar 14 '24 10:03 darkasevgen

There is another repeated label: "missile". According to huggingface the 2nd one has been changed to "projectile, missile".

EIFY avatar Mar 31 '24 22:03 EIFY

uniqueness doesn't really matter here... we and CLIP_benchmark are using the same class names and prompts as OpenAI did for their original CLIP evals. Changing that would alter the results and not be consistent with all other evals.

I'd also argue that the changes OpenAI made from default names are correct. If you look at the validation images, the curators did not understand the distinction .. both sunglass and sunglasses are sunglasses, and missiles are all missiles without much to distinguish any specific subtype correctly.

rwightman avatar Jun 07 '24 17:06 rwightman