DeepSpeed
DeepSpeed copied to clipboard
[BUG] Zero3 backward_hooks use the wrong interface
Describe the bug in file : deepspeed/runtime/zero/parameter_offload.py
` #Pre forward hook self.forward_hooks.append(module.register_forward_pre_hook(_pre_forward_module_hook))
# Post forward hook
self.forward_hooks.append(module.register_forward_hook(_post_forward_module_hook))
# Pre backward hook
self.backward_hooks.append(module.register_forward_hook(_pre_backward_module_hook))
# post backward hook
self.backward_hooks.append(module.register_forward_pre_hook(_post_backward_module_hook))`
why is backward_hooks using interface "module.register_forward_pre_hook" and "module.register_forward_hook" ?