REDN icon indicating copy to clipboard operation
REDN copied to clipboard

Is the score evaluation function correct?

Open kamwoh opened this issue 3 years ago • 0 comments

gold_count = len(gold_label)
for idx, s in enumerate(res):
    if idx in gold_label and s == 1:
        self.res[CORRECT] += 1
        self.triple_count_res[triple_count][CORRECT] += 1 if idx != self.na_id else 0
        self.without_na_res[CORRECT] += 1 if idx != self.na_id else 0
        self.na_res[CORRECT] += 1 if idx == self.na_id else 0
        self.normal_res[CORRECT] += 1 if is_normal_data and idx != self.na_id else 0
        self.multi_label_res[CORRECT] += 1 if is_multi_label_data and idx != self.na_id else 0
        self.over_lapping_res[CORRECT] += 1 if is_over_lapping_data and idx != self.na_id else 0
        gold_count -= 1
    if idx in gold_label:
        self.res[GOLD_POSITIVE] += 1
        self.triple_count_res[triple_count][GOLD_POSITIVE] += 1 if idx != self.na_id else 0
        self.without_na_res[GOLD_POSITIVE] += 1 if idx != self.na_id else 0
        self.na_res[GOLD_POSITIVE] += 1 if idx == self.na_id else 0
        self.normal_res[GOLD_POSITIVE] += 1 if is_normal_data and idx != self.na_id else 0
        self.multi_label_res[GOLD_POSITIVE] += 1 if is_multi_label_data and idx != self.na_id else 0
        self.over_lapping_res[GOLD_POSITIVE] += 1 if is_over_lapping_data and idx != self.na_id else 0
    if s == 1:
        self.res[PRED_POSITIVE] += 1
        self.triple_count_res[triple_count][PRED_POSITIVE] += 1 if idx != self.na_id else 0
        self.without_na_res[PRED_POSITIVE] += 1 if idx != self.na_id else 0
        self.na_res[PRED_POSITIVE] += 1 if idx == self.na_id else 0
        self.normal_res[PRED_POSITIVE] += 1 if is_normal_data and idx != self.na_id else 0
        self.multi_label_res[PRED_POSITIVE] += 1 if is_multi_label_data and idx != self.na_id else 0
        self.over_lapping_res[PRED_POSITIVE] += 1 if is_over_lapping_data and idx != self.na_id else 0
self.res[TOTAL] += 1   # shouldn't here to add "gold_count" instead of 1?
self.triple_count_res[triple_count][TOTAL] += 1 if ep[2] != self.na_id else 0
self.without_na_res[TOTAL] += 1 if ep[2] != self.na_id else 0
self.na_res[TOTAL] += 1 if ep[2] == self.na_id else 0
self.normal_res[TOTAL] += 1 if is_normal_data and ep[2] != self.na_id else 0
self.multi_label_res[TOTAL] += 1 if is_multi_label_data and ep[2] != self.na_id else 0
self.over_lapping_res[TOTAL] += 1 if is_over_lapping_data and ep[2] != self.na_id else 0

In your f1_metric.py, when calculating the score, as self.res[TOTAL] += 1 instead of gold_count, it is possible that your accuracy exceeded 100% (if gold_count >= 2). I wonder if my understanding is wrong?

kamwoh avatar Jan 28 '21 14:01 kamwoh