firefox-translations-training

firefox-translations-training copied to clipboard

This is a _very_ rough prototype for what's been discussed in https://github.com/mozilla/firefox-translations-training/issues/417 and https://github.com/taskcluster/taskgraph/issues/424. In this example, the decision about which tasks `merge-corpus` should depend on are being deferred and...

bhearsum

Bumps [grpcio](https://github.com/grpc/grpc) from 1.54.0 to 1.54.3. Release notes Sourced from grpcio's releases. Release v1.54.3 This is release 1.54.3 (gracious) of gRPC Core. For gRPC documentation, see grpc.io. For previous releases,...

dependabot[bot]

Dig deeper in sentences we translate badly to try and identify common failure themes

2

For example, by using tools such as [compare-mt](https://github.com/neulab/compare-mt), or by comparing translations with a “good” known one and sorting by BLEU/COMET.

marco-c

Override default opus-trainer config from experiment config

1

We'll want to experiment more with the modifiers and training data composition. Editing code adds friction to experimentation. The experiment config section might look like: ```yaml opus-trainer: teacher: stages: -...

eu9ene

gabrielBusta

[meta] Train harder to segment languages, like CJK languages

1

For harder to segment languages we have Chinese, Japanese, and Korean. We'll need to implement better tokenization support and segmentation support for these languages in order to train them. This...

gregtatum

Monolingual data has a word splitter that won't work for CJK

1

Right now it splits on word boundaries, and limits the size of the monolingual data to be less than 100 "words". This needs to be changed to support another segmentation...

gregtatum

Support training without a monolingual corpus

1

marco-c

We have a bunch of use cases when the DAG should be modified and we will have more in the future: 1. The number of chunks for translations 2. Skipping...

eu9ene

Switch on-push away from level 1 Taskcluster workers

1

We had a meeting with our security team to talk about the language training pipeline. We agreed that because we generate artifacts that eventually ship to users, that we would...

bhearsum