accelerate icon indicating copy to clipboard operation
accelerate copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

Accumulate the gradients of 2 models

Open lzqsd opened this issue 1 year ago • 3 comments

Thanks for the great work! I met an issue where I hoped to accumulate the gradients of 2 networks. How can I handle it with accelerator.accumulate() function? Looking forward to hearing from you!

lzqsd avatar May 11 '23 19:05 lzqsd

cc @muellerzr

sgugger avatar May 11 '23 19:05 sgugger

This is a limitation on PyTorch and we have a mildly working hack, once we've tested it a bit more independently + verify with PyTorch it's "expected" behavior we'll bring it into Accelerate directly

muellerzr avatar May 11 '23 20:05 muellerzr

Thanks for the quick response! Looking forward to it! I will close this issue once it is included.

lzqsd avatar May 11 '23 20:05 lzqsd