[Help Wanted] add Bleu scoring for validation

[vince62s]

I think this has been requested in the past, and before introducing early stopping based on Top N models not improving, it would be great to add this feature.

The shortest path in my view is to copy paste some code from sacrebleu since they do not expose a library. Of course, acknowledgement / licencing mention is required.

There are 3 functions to replicate: https://github.com/awslabs/sockeye/blob/master/sockeye_contrib/sacrebleu/sacrebleu.py#L951 https://github.com/awslabs/sockeye/blob/master/sockeye_contrib/sacrebleu/sacrebleu.py#L999 https://github.com/awslabs/sockeye/blob/master/sockeye_contrib/sacrebleu/sacrebleu.py#L1022 (and dependencies)

@coder1729 can you take care of this ?

[coder1729]

Sure, i'll try it !

[coder1729]

@vince62s will a sentence level bleu for each sentence in the training batch and printing bleu along with other stats like accuracy work for you? It will be need to be modified (not just copied) to ignore pad tokens in n grams when splitting sentences . Additionally it should be added in validation as well, correct? I can go ahead once you confirm. Thanks !

[vince62s]

Well, I think for now, we just need to calculate the corpus_bleu on the validation set. But please include sentence_bleu in the library, it could be useful for future work. Training bleu could also be useful for tensorboard visualization but let's start with validation first. (this is more a criterion for early stopping / best models storing). Thanks.

[erip]

Sacrebleu is exposed as a library on pypi, but it doesn't seem to be available through a conda channel.