sdtblck

> In addition to `to_sequential`, there may be another way we could accomplish this while keeping the normal `PipelineModule`, if that would be useful. > > If we short-circuit this...

> This is a great idea, thanks @sdtblck ! > > One caveat is that we lose the activation checkpointing that the `PipelineModule`'s forward can be configured to use. But...

Hi @ShadenSmith I think the two latest commits should fix both the above requirements. There is maybe some repeated code between `SequentialModel` and `PipelineModule` that could be slimmed down -...

ah ok, I think i misunderstood what the 'add_prefix_space' option does. I had assumed it controlled whether the space between words was at the beginning, or end of the token....

Ok @sweinbach I did some pretty extensive testing / debugging of this, and had to change a fair few things to get it to work at all, but I think...

@sweinbach anything else to add, or would you say this is ready to merge?

Aside from the above ^ lgtm :rocket:

> > I'm not sure of the motivation for the changed to fused_kernels.py Also, I really don't like the requirements being so granular like this. I don't see the need...

@hyunwoongko actually in neox we also load onto the CPU and then move to the GPU, so i'm not sure this is a problem

Hey @zhuzilin , really interesting! firstly, wrt to the speed difference between pp=0 and pp=1, we also found a similar thing, see https://github.com/EleutherAI/gpt-neox/pull/269 . Although maybe the speed difference isn't...