spupyrev

i'm not entirely sure what you mean, that's been a while since I worked on this. In any case, feel free to modify the code and send a diff :)

Tests and measurements are done on an older rev (467dc1bcd40804611fa79cb50db8211b7d1d891e), since the latest has some tracing issues (e.g., https://github.com/pytorch/PiPPy/issues/1087)

> Nice work! Could you please provide some readings/explanations regarding this implementation? This is an original work but the ideas are fairly well-known. Perhaps the closest implementation are works [here](https://arxiv.org/pdf/2311.03703)...

The latest commit makes the computation much faster (e.g., ~2sec on gpt2) by "pre-solving" the instance and assigning some nodes to specific stages. The code now tested and works on...

New measurements on the latest revision: | model | ngpus | manual | (old-)autosplit | (new-)autohsplit | algo time | |--------|--------|--------|--------|--------|--------| | pippy_bert | 2 | 0.0876 | 0.1030 |...