How do I get the same human ranking as your paper?

[gotutiyan]

Hello,

I want to get the human ranking and its concrete scores of your work. So I followed your instruction (2. Run TrueSkill) like this:

#!/bin/bash
set -eu

if [ ! -e gec-ranking ]; then
    git clone https://github.com/cnap/gec-ranking.git
fi

if [ ! -e wmt-trueskill ]; then
    git clone https://github.com/keisks/wmt-trueskill.git
fi

cd wmt-trueskill
if [ ! -e result ]; then
    mkdir result
fi

pip2 install -r requirements.txt
cd src
cat ../../gec-ranking/data/all_judgments.csv | python2 infer_TS.py ../result/gec -n 2 -d 0 -s 2

And got the result as follows (note that I sorted them by its score):

[('CAMB', 0.10534599231875195),
 ('AMU', 0.04306970884724038),
 ('RAC', 0.020777579012258046),
 ('src', 0.002039173907479636),
 ('UFC', -0.010643493294255905),
 ('IITB', -0.06176253371780177),
 ('CUUI', -0.0891020502995365),
 ('POST', -0.10582097317332297),
 ('SJTU', -0.11394275778296624),
 ('UMC', -0.18228045159921868),
 ('PKU', -0.19595280718488897),
 ('IPN', -0.2648197969906597),
 ('NTHU', -0.3377111687837812)]

However, the human ranking mentioned in Table 2 in your paper is not identical to the above ranking.

How do I get the same human ranking as your paper?

Thank you.

[pygongnlp]

Hi,

If you have got the right human rankings? I got another different ranking based on wmt-trueskill, which also wrong with in this paper