Evgeny Pavlov

Hi, what's the status of the en-fr model training? At what stage in the pipeline is it now?

I also discovered this issue while retraining my models. It was fixed a month ago https://github.com/mozilla/firefox-translations-training/blob/22a3751a09dfdb2ba52f4d08c285e424c533dcde/configs/config.prod.yml#L64 I created an issue https://github.com/mozilla/firefox-translations-training/issues/78 to stop training if the quality is too low.

There is also another issue https://github.com/mozilla/firefox-translations-training/issues/75. I don't know how your training is configured, but when I was retraining ru,pt,it -> en models, the quality of neither fine-tuned teacher was...

> EFS might be too slow for mmap option That's what I suspected. I tried `QdrantClient(..., timeout=100)` and it works for a bit and then starts failing with `Broken pipe`....

I think we're still struggling with a bunch of issues, so I'd add them here and keep this one open until we can confidently train new languages without breakage.

One small correction: `pipeline/train/spm-vocab.sh` trains a vocab and we don't want to track this in W&B. So only `train.sh` should be patched.

> is this going to be covered by the tracking platform? No, we should store the artifacts on GCS regardless of the solution for experiment tracking. In MLFlow case, you...

@gabrielBusta > Do we need a dedicated bucket? I would prefer a dedicated bucket for the translation artifacts. Also, I think it's a best practice not to mix things because...

Let's sync this work with #312. We'll need to download and publish old artifacts to W&B but what if they become unavailable?

@bhearsum we have a list of old experiments here: https://github.com/mozilla/firefox-translations-training/issues/312#issuecomment-1946802157