sanketpurandare

Allred

2

Added BertLarge model that has 16 attention heads, 24 hidden layers and 1024 as hidden size. Added ddp_trainer as an example that allows to benchmark ddp, all_reduce annd non_ddp version...

First draft FSDP2 Memory Tracker

3

Introducing dispatch based memory tracker for tracking FSDP2 memory and module-wise memory breakdown. cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol...

Memory Tracker for tracking Module wise memory

2

A tool for measuring module wise memory consumption. It also provides a breakdown of memory consumption in terms of parameters, gradients, activations and optimizer state. Works in fake tensor mode...

Runtime Estimator for estimating GPU compute time

8

This PR adds a basic Runtime Estimator for single-device models. It estimates the GPU runtime in milliseconds using various estimation methods under the ``FakeTensorMode``. It provides a ``TorchDispatchMode`` based context...