RuntimeError: copy_if failed to synchronize: the launch timed out and was terminated

[jaeyun95]

trafficstars

i have a problem like this: RuntimeError: copy_if failed to synchronize: the launch timed out and was terminated

Traceback (most recent call last): File "train.py", line 125, in loss = output_dict['loss'].mean() + output_dict['cnn_regularization_loss'].mean() File "/home/ailab/anaconda2/envs/r2c/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/ailab/r2c/models/multiatt/model.py", line 156, in forward obj_reps = self.detector(images=images, boxes=boxes, box_mask=box_mask, classes=objects, segms=segms) File "/home/ailab/anaconda2/envs/r2c/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/ailab/r2c/utils/detector.py", line 112, in forward box_inds = box_mask.nonzero() RuntimeError: copy_if failed to synchronize: the launch timed out and was terminated

When I was training this model, it is stop ! what can i do?

env titan X cuda 9.0

[rowanz]

Sorry, i've never encountered this error while training the model a variety of setups.

There could be a few things going on. one thing to double check - are you using my exact environment with pytorch 1.0?