sacrebleu
sacrebleu copied to clipboard
mjpost
Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
I am attempting to compute chrF++ for a set of predictions and references. If I use `sacrebleu` cli (`sacrebleu ref.eng_Latn.tok < pred.eng_Latn.tok -m bleu chrf --chrf-word-order 2`), I find a...
Description: Added a one-line optimization for initialization of the `precissions` list in `def compute_bleu` function. Motivation: I have to do a total of 7 billion computations so I was looking...
Currently, when you calculate BLEU with different `max_ngram_order`'s and everything the same, they will have the same signature when you use `bleu_metric.get_signature().format(short=True)`. Something like `#:1|c:mixed|e:no|tok:13a|s:exp|v:2.3.1`. Should an argument be added...
Is there an implementation of Human-mediated Translation Edit Rate (HTER) algorithm? Related paper: https://aclanthology.org/2006.amta-papers.25/
`In [1]: from sacrebleu.metrics import BLEU, CHRF` ` ...: refs = [["请关掉灯光。"]]` ` ...: sys = ["请关闭灯光。"]` `In [2]: bleu = BLEU(trg_lang="zh")` `In [3]: bleu.corpus_score(sys, refs)` `Out[3]: BLEU = 37.99...
Hello, Thank you for your contribution with this library. I am experimenting a problem computing BLEU and CHRF when some hypothesis are empty strings. **The code to reproduce the problem...
Description: I am encountering an issue with SacreBLEU where I am getting inconsistent scores between a loop implementation and a separate check for individual translations. Here are the details of...
Related to #225 This is caused by that the "translator" keywords are in "src" instead of "ref" sometimes, "wmt21/dev is-en" and "wmt22 liv-en" for example.
This line outputs the following message below which might confuse the user to think that flores101 tokenizer is activated even when the tokenizer flores200 is selected. Better trigger the warning...
