lightning-thunder
lightning-thunder copied to clipboard
[benchmark] migrate to fsdp/ddp after jit, from fsdp/ddp before jit
Llama-2-7b-hf & fsdp
container of 20240804. 8 H100 80GB HBM3.
command: torchrun --nproc_per_node=8 thunder/benchmarks/benchmark_litgpt.py --compile=thunder_inductor_cat --model_name=Llama-2-7b-hf --distributed_mode=fsdp --shard_mode=zero2 --bucketing_mode=none
| main 5cc3011 | pr 3869547 | |
|---|---|---|
| token/s | 85765.16 | 85695.95 |
| tokens/s/GPU | 10720.65 | 10711.99 |
| Memory | 42.06 GB | 42.06 GB |
cc @mpatel31415
related:
- https://github.com/Lightning-AI/lightning-thunder/issues/478