Evgeny Pavlov

I'm not sure if it's the same issue or not but I have to run cancel all action 2-3 times before I can see in UI that all the tasks...

It's ready for review, but I just launched the experiment that will take a while to complete: https://firefox-ci-tc.services.mozilla.com/tasks/groups/Uv6EgA9SQdGT54nkWFwACQ

> > https://firefox-ci-tc.services.mozilla.com/tasks/Jmrh6CCOS_23BG-dNKUC9A > > I see this ultimately failed because of what appears to be a bunch of spot terminations killing the `alignments-teacher` task. Do you want to switch...

> > > https://firefox-ci-tc.services.mozilla.com/tasks/Jmrh6CCOS_23BG-dNKUC9A > > > > > > I see this ultimately failed because of what appears to be a bunch of spot terminations killing the `alignments-teacher` task....

We investigated with @bhearsum and it seems the issue is OOM. I've been monitoring it through interactive task and the memory is steadily growing and reached 62% after several hours....

Everything that starts from "evaluate": https://github.com/mozilla/firefox-translations-training/blob/main/taskcluster/kinds/evaluate-quantized/kind.yml https://github.com/mozilla/firefox-translations-training/blob/main/taskcluster/kinds/evaluate-teacher-ensemble/kind.yml https://github.com/mozilla/firefox-translations-training/blob/main/taskcluster/kinds/evaluate/kind.yml The goal is to integrate everything we had in the offline publisher. Also, we should reuse the code from the offline publisher...

UpperCase/TitleCase augmentations are supposed to fix the problems related to casing, but we still need to test the final model in Nightly to verify.

I think we should not use those legacy scripts and use only OpusCleaner. It would make sense to enable it by default at this point.

Yes, title case augmentation is supposed to take care of the example you posted.

I use tmux and just refresh the page and then attach to tmux session again. I think it's good enough. We should document this behavior though.