PiPPy
PiPPy copied to clipboard
pytorch
Pipeline Parallelism for PyTorch
1. Create a branch `hf_example_qa` 2. Copy [question-answering](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) dir to [hf](https://github.com/pytorch/PiPPy/tree/main/examples/hf) 3. Add files, commit, create a PR(`hf_example_qa`->`main`) 4. Create another branch `hf_example_qa_pippy` on top of `hf_example_qa` and do all...
the original [run_glue.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py): ``` ***** train metrics ***** epoch = 3.0 train_loss = 0.4244 train_runtime = 0:20:44.02 train_samples = 3668 train_samples_per_second = 8.846 train_steps_per_second = 0.277 ***** eval metrics *****...
Create hf_test_utils.py and move HFGPT2Tracer, HFBertTracer and other HF-related code there
This is a follow up for https://github.com/pytorch/PiPPy/pull/363 @jamesr66a please add the details
We have only 3 example of full training loop end-to-end examples: 1. MNIST example 2. ResNet50 example 3. T5 translation example(in separate repo). We should move it to this repo!...
Context: Text summarization evaluation step requires model to have generate method: ``` File "/Users/pbelevich/PycharmProjects/PiPPy/pippy/hf/utils.py", line 229, in run_worker run_master(model_args, data_args, training_args, pp_ranks_per_dp_group[rank]) File "/Users/pbelevich/PycharmProjects/PiPPy/examples/hf/summarization/run_summarization.py", line 745, in run_master metrics =...
use it to exclude files instead of [grep -v](https://github.com/pytorch/PiPPy/blob/dd71b6407ef7f59867a5de66fab21f746f2cb873/.github/workflows/code-quality.yml#L35) https://mypy.readthedocs.io/en/stable/config_file.html
Possible scenarios: * Compare plain PiPPy with plain Megatron * May be hard to implement * ~~Compare HF [MegatronBERT](https://huggingface.co/docs/transformers/model_doc/megatron-bert) with PiPPy-wrapped HF [BERT](https://huggingface.co/docs/transformers/model_doc/bert)~~ * ~~Choose a task (ForMaskedLM, ForCausalLM, ForNextSentencePrediction,...