Botao Chen

Torch profiler was added as an optional component in https://github.com/pytorch/torchtune/pull/627 and we show case how to use it in lora_finetune_single_device.py recipe which won't have this issue. To address this, we...

Thanks for making this RFC PR @rohan-varma! Share my 2c - Seems there are many functions that support different tracking use cases. Shall we begin from less generic class design...

> @SLR722 It makes sense to me to make the profiler and memory snapshot as individual standalone components, though I don't see why these should be together within the same...

LGTM! Please fix the tests and clean up the code and then we are good to go!

> Also let me know what thoughts you have on extending to other recipe tests. I think with llama3 recipe test as an example, it's easy to expand to other...

Thanks for introducing DoRA to torchtune! - could you add an example distributed yaml config to showcase DoRA? - In DoRA paper, DoRA outperformances LoRA on several tasks while they...

I made several updates to this PR to push it closer to the finish line. - fix and clean up several unit tests to let majority of the unit tests...

To further verification the correctness of this DoRA work - compare the loss between LoRA and DoRA - LoRA: [0.8608](https://wandb.ai/torchtune_dev/torchtune/reports/loss-24-08-26-21-08-15---Vmlldzo5MTU4NTY3) - DoRA: [0.8631](https://wandb.ai/torchtune_dev/torchtune/reports/loss-24-08-26-21-09-51---Vmlldzo5MTU4NTg0) - compare the correlation between changes in...