testing on NYT

[hemmatan]

Hi! Since the NYT data is built through Distant Supervision, I guess that it has to be noisy, and using it as a test set does not lead to a correct evaluation. Then, how do you evaluate on NYT? Is there a part of NYT that is labeled manually?

[131250208]

@hemmatan Hi, thanks for your interest. We just followed the previous SoTA and used the same datasets. For all I know, they are not labeled manually and no manually labeled NYT was used before. Yes, this may lead to unreliable evaluation. If you want to test the model in a more reliable way, you have to label them by yourself.

[hemmatan]

@131250208 Thank you very much :)