New Featuers for KGExplainaBoard

[pfliu-nlp]

• [x] Generalize evaluation metric: Hit@k (k is a dynamic parameter)

  • a simple way to do this we can support a variety of metric variants, such as hit@1, hit@2, hit@3, hit@5, for example, summarization tasks support many metrics: https://github.com/neulab/ExplainaBoard/blob/8dcc5885d4a9523bb33c3d55a6d8372ba2ef07d2/explainaboard/tasks.py#L50

• [ ] Support a dynamic number (n_bucket) of buckets,

  • I think we could make n_bucket as a member of SysOutputInfo class, which then can be specified by users via manipulating metadata inside the processor The downside is that all features share the same value of n_bucket.
  • We also can make the bucket_info in the processor customizable

• [x] Customized sort of buckets

  • Similar to the above one, and I think this is the relevant function script

• [ ] How to utilize the prediction score (e.g., probability)

  • do we need to consider the calibration features, and draw a similar figure as shown in figure 1

• [ ] Generalize task type: from tail prediction to link prediction

  • do we need new system output files?

• [x] upload hyper-parameter for each system?