CMU-MOSEI Labeling Criteria

[MohdElgaar]

Hi.

What criteria do you use for creating the binary emotion labels?

Suppose a video has been rated by 10 MTurk workers, with the following average score: [happiness, sadness, fear, surprise, anger, disgust] -> ['0.40', '0.07', '0.00', '0.07', '0.03', '0.27']

What would the label be, according to the definition you used in the original paper and the Graph-MFN experiments? I also looked at other recent papers that did (multi-label) emotion recognition on CMU-MOSEI but I couldn't find the criteria for labeling.

Here are a few actual scores I just extracted from the dataset as an example:

Video_ID: TxRS6vJ9ak0
Number of ratings: 21
Scores: ['1.95', '0.00', '0.00', '0.00', '0.00', '0.00']
Video_ID: JW2HHfQiGVs
Number of ratings: 18
Scores: ['0.06', '0.22', '0.00', '0.00', '0.06', '0.00']
Video_ID: 22335
Number of ratings: 21
Scores: ['0.06', '0.22', '0.00', '0.00', '0.06', '0.00']
Video_ID: CbQxC1iPyS8
Number of ratings: 9
Scores: ['0.56', '0.11', '0.11', '0.22', '0.00', '0.00']
Video_ID: K_5u2Wh_wGk
Number of ratings: 21
Scores: ['0.14', '0.38', '0.05', '0.10', '0.05', '0.00']
Video_ID: gzZ8NTeTzRs
Number of ratings: 6
Scores: ['0.17', '0.50', '0.00', '0.00', '0.17', '0.00']
Video_ID: 9-EO2oCAGA4
Number of ratings: 21
Scores: ['0.17', '0.50', '0.00', '0.00', '0.17', '0.00']
Video_ID: BseI8-TYWm8
Number of ratings: 6
Scores: ['2.00', '0.00', '0.00', '0.00', '0.00', '0.00']
Video_ID: QoIsjc4-GIg
Number of ratings: 30
Scores: ['0.40', '0.07', '0.00', '0.07', '0.03', '0.27']

Thank you.

[A2Zadeh]

Hi @MohdElgaar,

Happy to help. Are you referring to the annotation criteria or our criteria on multi-class classification?

[MohdElgaar]

@A2Zadeh I was referring to the criteria for annotation. How do you convert the MTurk survey results into actual annotations? Each video is annotated by multiple workers. So how do you determine if an emotion is present in a video? Is it if at least 1 worker votes that it is present? Or 50% of workers vote that it is present? Or is it based on the scores (0~3), e.g. if the average score is higher than 1?

[ajwaaslam]

@A2Zadeh how can I get the labels for each modality? as I have to make unimodals for every modality. Kindly, do let me know