[meta] issues blocking us from using spot instance for training tasks

[bhearsum]

We turned off spot instances for training tasks in #356. This issue tracks all the things we need to do before we can turn them back on.

### Blocks turning spot instances back on
- [ ] https://github.com/mozilla/firefox-translations-training/issues/400
- [ ] https://github.com/mozilla/firefox-translations-training/issues/270
- [ ] https://github.com/mozilla/firefox-translations-training/issues/164
- [ ] https://github.com/mozilla/firefox-translations-training/issues/515

### Investigate after spot instances are turned back on
- [ ] https://github.com/taskcluster/taskcluster/issues/6685
- [ ] https://github.com/mozilla/firefox-translations-training/issues/271

[eu9ene]

It's important to make sure we'll be able to track experiments properly. It's probably better not to split training until we have real-time publication. Even then we'll need to support reuploading old training runs that were split.

[bhearsum]

I believe we're now in a place to reliably train on spot instances. Should we close this out and unlink the remaining open issues?

[bhearsum]

I believe we're now in a place to reliably train on spot instances. Should we close this out and unlink the remaining open issues?

I'm going to call this done, as nothing else blocks. Feel free to re-open.