How to implement 1F1B pipeline parallelism in Jax?

Open MoFHeka opened this issue 1 year ago • 1 comments

Not GPipe. Run pipeline forward meanwhile backward.

Jul 08 '24 16:07 MoFHeka

We are still looking into this in the open source side! Likely at least 6 months away

Aug 28 '24 02:08 gobbleturk

Is this still in progress for maxtext?

Dec 09 '24 17:12 sbhavani

Yes this is still something we are considering, likely requires a paradigm shift (multiple programs multiple data) to support

Dec 09 '24 18:12 gobbleturk