llm-foundry
llm-foundry copied to clipboard
Upgrade to `mosaicml-streaming==0.5.x`
Yup will run some convergence tests before merging
Would recommend waiting on this merge until we get a hotfix release 0.5.1
With baseline usage, main and this branch have similar performance
MFU is also effectively exactly the same
the only diff is that the update uses slightly less sys mem
https://wandb.ai/mosaic-ml/streaming051_updt