Brian Deuksin Kwon

@Shikib Thank you for a quick reply and detailed explanation for my questions. 1. As suggested in the paper for removing outliers in five annotations given a question, I applied...

> You mentioned that you was able to reproduce the same result of the average score. How could you do that? I tried to reproduce results based on the original...

Also, in the original code, It seems that the logic for removing outliers is different from that in the paper. In the paper, **half of the standard deviation** of the...

@Shikib I am sorry. I had a mistake to report the results that I gained. Based on your codes and data files, I just ran a python file (analyze.py). But,...

It's very strange. I downloaded data files again (https://drive.google.com/file/d/1-pMlmO6s0qlr5UpDf0_WKyjawck_bHv6/view?usp=sharing) and ran the analyze script inside the directory just a while ago. But the result is the same as the previous...

@Shikib To double check if there was a mistake in reproducing the results based on your guidance with the file and codes (https://drive.google.com/file/d/1-pMlmO6s0qlr5UpDf0_WKyjawck_bHv6/view?usp=sharing), I also asked my colleague to reproduce...

@Shikib Thank you for following up my issues. According to your comment, I've changed the code (128 line, analyze.py) into what you mentioned (`system_scores[system_map[e]].append(np.mean(mean_scores[3*i:3*(i+1)]))`). However, I still could not obtain...