sacrebleu
sacrebleu copied to clipboard
Published
20 hours ago •
mjpost
mjpost
BLEU and chrF2 on Chinese
In [1]: from sacrebleu.metrics import BLEU, CHRF
...: refs = [["请关掉灯光。"]]
...: sys = ["请关闭灯光。"]
In [2]: bleu = BLEU(trg_lang="zh")
In [3]: bleu.corpus_score(sys, refs)
Out[3]: BLEU = 37.99 83.3/60.0/25.0/16.7 (BP = 1.000 ratio = 1.000 hyp_len = 6 ref_len = 6)
In [4]: bleu.get_signature()
Out[4]: nrefs:1|case:mixed|eff:no|tok:zh|smooth:exp|version:2.3.1
In [5]: chrf = CHRF()
In [6]: chrf_score = chrf.corpus_score(sys, refs)
In [7]: print(chrf_score)
Out[7]: chrF2 = 28.06
BLEU score 37.99 and chrF2 score 28.06, these two values do not really reflect the close relationship between refs and sys. What the more appropriate usage on Chinese?
