Evgeny Pavlov

please sync with main, it includes a simple CI now

As a temporary workaround you can either increase retries (`--restart-times 3`) or increase the number of threads for this job or both. Parallelization is controlled with `--cores all` option and...

@XapaJIaMnu could you please clarify how this will help?

This might be a bug of Marian. Memory shouldn't grow after `--shuffle-in-ram` is removed and we should use `--shuffle data` mode. It was discussed in https://github.com/mozilla/firefox-translations-training/pull/70#discussion_r800975032

Training teachers with `--shuffle batches` leads to such training curves. Maybe other factors are at play here.

Related Marian issue: https://github.com/marian-nmt/marian-dev/issues/148

Increasing early stopping thresholds can help, but it still does not properly fine tune on some languages, I assume because of low quality of the data. ``` training-teacher-base: # remove...

This issue happens because of a conflict with preinstalled conda. We should modify the Makefile so that it always uses our mamba installation for snakemake.

> One open question here is what Snakemake does with non-0 exit codes of scripts. I am completely new to Snakemake ... Snakemake deletes the job output that is specified...

Also use `--disable_hardrules` as it was suggested [here](https://github.com/mozilla/firefox-translations-training/issues/71#issuecomment-1040029744).