RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

[DWBSIC]

Traceback (most recent call last): File "segmentation_train.py", line 117, in main() File "segmentation_train.py", line 69, in main TrainLoop( File "E:\MedSegDiff-master2.0\guided_diffusion\train_util.py", line 83, in init self._load_and_sync_parameters() File "E:\MedSegDiff-master2.0\guided_diffusion\train_util.py", line 139, in _load_and_sync_parameters dist_util.sync_params(self.model.parameters()) File "E:\MedSegDiff-master2.0\guided_diffusion\dist_util.py", line 78, in sync_params dist.broadcast(p, 0) File "D:\anaconda3\envs\py38\lib\site-packages\torch\distributed\distributed_c10d.py", line 1438, in wrapper return func(*args, **kwargs) File "D:\anaconda3\envs\py38\lib\site-packages\torch\distributed\distributed_c10d.py", line 1561, in broadcast work.wait() RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

您好，我在Linux系统中进行训练没有任何问题，但是我尝试在Windows系统下进行训练，所有的参数配置都是按照作者在readme中的参数设置，请问为什么会出现上述的错误，请大家帮助解决。

Hello, I have no problem with training in Linux system, but I try to train in Windows system, and all parameters are set according to the author's parameter Settings in readme. May I ask why the above errors occur, please help to solve.

if you want to chat with me: WeChat: DWBSIC

[170744039]

我也遇到了这个问题，因为我是windows，dist.init_process_group(backend="gloo", init_method="env://")，所以初始化中的backend首先需要修改，不然就会报ncll错误。然后就遇到了跟您一样的问题，但是这样修改后可以跑通，在dist.broadcast前加入 p = p +0 就可以了。我用的是自己的训练数据，但是现在的问题是GPU 利用率非常低，只能达到六七十的样子。请问您完美解决这个问题了吗

[xupinggl]

It has been solved according to your method. Thank you very much! @170744039

[blackcat1121]

@170744039 Thank you for the solution! I have so limited knowledge of DDP that I have no clue how to fix it even though I know its probably a Windows/Linux problem. Could you share more how p = p +0 solves "a leaf Variable that requires grad is being used in an in-place operation"?

[WuJunde]

我也遇到了这个问题，因为我是windows，dist.init_process_group(backend="gloo", init_method="env://")，所以初始化中的backend首先需要修改，不然就会报ncll错误。然后就遇到了跟您一样的问题，但是这样修改后可以跑通，在dist.broadcast前加入 p = p +0 就可以了。我用的是自己的训练数据，但是现在的问题是GPU 利用率非常低，只能达到六七十的样子。请问您完美解决这个问题了吗

@170744039 Hi! Thank you for your awesome solution for Windows. Could you help to pull a request for your modification? I do not have Windows-installed pc, so it would be great if someone could help its windows extensibility.

[WuJunde]

我也遇到了这个问题，因为我是windows，dist.init_process_group(backend="gloo", init_method="env://")，所以初始化中的backend首先需要修改，不然就会报ncll错误。然后就遇到了跟您一样的问题，但是这样修改后可以跑通，在dist.broadcast前加入 p = p +0 就可以了。我用的是自己的训练数据，但是现在的问题是GPU 利用率非常低，只能达到六七十的样子。请问您完美解决这个问题了吗

@170744039 Hi! Thank you for your awesome solution for Windows. Could you help to pull a request for your modification? I do not have Windows-installed pc, so it would be great if someone could help its windows extensibility.

[Longchentong]

dist_util.py

old

def sync_params(params): """ Synchronize a sequence of Tensors across ranks from rank 0. """ for p in params: with th.no_grad(): dist.broadcast(p, 0)

new

def sync_params(params): """ Synchronize a sequence of Tensors across ranks from rank 0. """ for p in params: p=p+0 with th.no_grad(): dist.broadcast(p, 0)