RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88

[Joge-jiaye]

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 170, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 394, in _make_request conn.request(method, url, **httplib_request_kw) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1107, in request self._send_request(method, url, body, headers) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1152, in _send_request self.endheaders(body) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1103, in endheaders self._send_output(message_body) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 934, in _send_output self.send(msg) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 877, in send self.connect() File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 200, in connect conn = self._new_conn() File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 182, in _new_conn self, "Failed to establish a new connection: %s" % e urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 756, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/visdom/init.py", line 711, in _send data=json.dumps(msg), File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/visdom/init.py", line 677, in _handle_post r = self.session.post(url, data=data) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 590, in post return self.request('POST', url, data=data, json=json, **kwargs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused',)) [Errno 111] Connection refused create web directory ./checkpoints/enlightening/web... THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=88 error=10 : invalid device ordinal Traceback (most recent call last): File "train.py", line 31, in model.optimize_parameters(epoch) File "/home/fjy/work/EnlightenGAN/models/single_model.py", line 394, in optimize_parameters self.forward() File "/home/fjy/work/EnlightenGAN/models/single_model.py", line 244, in forward self.fake_B, self.latent_real_A = self.netG_A.forward(self.real_img, self.real_A_gray) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 69, in forward inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 80, in scatter return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 38, in scatter_kwargs inputs = scatter(inputs, target_gpus, dim) if inputs else [] File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 31, in scatter return scatter_map(inputs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 18, in scatter_map return list(zip(*map(scatter_map, obj))) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 15, in scatter_map return Scatter.apply(target_gpus, None, dim, obj) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/_functions.py", line 74, in forward outputs = comm.scatter(input, ctx.target_gpus, ctx.chunk_sizes, ctx.dim, streams) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/cuda/comm.py", line 188, in scatter with torch.cuda.device(device), torch.cuda.stream(stream): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/cuda/init.py", line 209, in enter torch._C._cuda_setDevice(self.idx) RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88