Publish training charts from Taskcluster

[eu9ene]

This includes publishing:

• live training logs to W&B dashboards

I assume we'll have separate publishing scripts for other things.

Let's use Taskgraph transforms not to pollute Taskcluster kinds with even more logic.

[La0]

We can start working on this now.

I identified these next steps to discuss together:

1. build a dedicated command in the tracking package to support specific tasks (training & evaluation) and use a Taskcluster secret to get the Weight & Biases token
2. Setup the publication code through a Taskcluster transform
3. Patch training shell scripts to capture output and publish from logs
   • when an env variable is set
   • scripts to patch
     • pipeline/train/train.sh
     • pipeline/train/spm-vocab.sh
4. Identify other tasks to track & publish

[La0]

After discussion with Evgeny, we can start to build the dedicated script to interact with Taskcluster secret & logs.

[eu9ene]

One small correction: pipeline/train/spm-vocab.sh trains a vocab and we don't want to track this in W&B. So only train.sh should be patched.

[La0]

@evabardou will work on Taskcluster secret support.

We can start experimenting log parsing from CI using --in-stream option too