running eval_multipro.py error:cuda runtime error

[sunwillz]

trafficstars

Hi, when i finished my training, i run the eval_multipro.py and get below error: nr_dev: 1 0%| | 0/5145 [00:00<?, ?it/s]process:0, start_idx:0, end_idx:5145 THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=34 error=3 : initialization error Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "eval_multipro.py", line 129, in worker torch.cuda.set_device(dev_id) File "/home/sunyangwei/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 264, in set_device torch._C._cuda_setDevice(device) RuntimeError: cuda runtime error (3) : initialization error at /pytorch/torch/csrc/cuda/Module.cpp:34

could you please tell me how to solve it? I run it with pytorch 1.0 on ubuntu16.04

[Jongchan]

Change the starting of eval_multipro.py as, then it should work. I use Ubuntu 16.04 and PyTorch 1.0.1

# System libs
import os
from tqdm import tqdm
import datetime
import argparse
from distutils.version import LooseVersion
#from multiprocessing import Queue, Process
# Numerical libs
import numpy as np
import math
import torch
import cv2
import torch.nn as nn
from scipy.io import loadmat
import torch.multiprocessing as mp
from torch.multiprocessing import Queue, Process
mp.set_start_method('spawn', force=True)

[zhihengli-UR]

You can also add multiprocessing.set_start_method('spawn') following if __name__ == '__main__':