accelerate
accelerate copied to clipboard
Only wrap modules in DDP if they require grad
Like the title says. Fixes #760.
There is just such a check in DistributedDataParallel, which I do the opposite of.
I would like to add a test so that this feature doesn't regress, but it's not clear to me how to add a test.
I could add a module with only buffers and add it to the prepare() call on line 78, then check that the device is the accelerator's device.
https://github.com/huggingface/accelerate/blob/693d46826e32507376d44f99967df4710886c984/tests/test_kwargs_handlers.py#L73-L79
The documentation is not available anymore as the PR was closed or merged.