DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard

[BUG] Zero3 backward_hooks use the wrong interface

Open zhxfl opened this issue 1 year ago • 1 comments

Describe the bug in file : deepspeed/runtime/zero/parameter_offload.py

` #Pre forward hook self.forward_hooks.append(module.register_forward_pre_hook(_pre_forward_module_hook))

    # Post forward hook
    self.forward_hooks.append(module.register_forward_hook(_post_forward_module_hook))

    # Pre backward hook
    self.backward_hooks.append(module.register_forward_hook(_pre_backward_module_hook))

    # post backward hook
    self.backward_hooks.append(module.register_forward_pre_hook(_post_backward_module_hook))`

why is backward_hooks using interface "module.register_forward_pre_hook" and "module.register_forward_hook" ?

zhxfl avatar Aug 30 '23 11:08 zhxfl