Dataset and mapping to the indexes

[karunakc]

Is it possible to get the mapping of the indexes in benchmark.json, etc to the actual dataset used that was mentioned in the paper appendix? Although some data points are available in the benchmark.json, we need to perform the final classification task. Thank you!

[ruiqi-zhong]

Sorry that somehow the table in the Appendix was not sorted in order and I do not currently have the index mapping.

However, feel free to take a look at benchmark_sec_4/manual_evaluation.json. If the description in my paper is accurate, you can find the "descriptions by our system" column but looking at the results for system 1 and taking the descriptions that have the highest rating.

[karunakc]

Thank you for the quick response! I was referring to Appendix Section F which cited the sources for the binary task. We required the mapping of the datasets mentioned in section F to the human annotations/annotations generated by the system. Although there were positive and negative samples in benchmark.json along with different annotations in manual_evaluation.json, we weren't able to identify all 54 binary classification tasks + the datasets from Appendix Section F.

Thank you!