torchtune
torchtune copied to clipboard
Debug flag to quickly run and test configs
We should introduce a debug mode at the CLI level that will automatically run a config on CPU without distributed just for a small number of steps/epochs. This is really useful for quick sanity checks and testing purposes, and may be needed for recipe integration tests.
should this just be --device cpu
?
This will need some more work than just --device cpu
since our current recipes have fsdp very tightly integrated. That said we have single device recipes which should land in the next week and those should work OOTB on CPU. cc: @rohan-varma, @ebsmothers
It will be good to have a debug flag for both CPU and GPU for sanity testing the setup is working fine for the different type of configs