en-tr translation quality feedback

[selimsum]

I've been playing around with en-tr translations and I'd like to share some feedback.

I chose this story for a detailed comparison with Google Translate.

In the Google docs linked below, I've highlighted each sentence with colors corresponding to four levels of translation quality. Obviously, these are my own interpretations.

Our model also generated some made-up words (13 out of 800 words). I've marked them as well.

It's quite interesting that our COMET score turns out to be within 5% of Google Translate's COMET score. I would rate our translation quality much lower than that. And this makes me think that COMET shouldn't be considered a reliable metric for Turkish.

In my opinion, our model's current translation quality in general is not good enough to be shipped yet.

The devil's in the details - Firefox Translations The devil's in the details - Google Translate