ERROR:Call to call_cuIpcOpenMemHandle results in CUDA_ERROR_ALREADY_MAPPED

[randyzwitch]

trafficstars

Every other select_ipc_gpu call, getting this error. It looks like we might not be releasing the IPC handle correctly.

ERROR:Call to call_cuIpcOpenMemHandle results in CUDA_ERROR_ALREADY_MAPPED
Traceback (most recent call last):

  File "<ipython-input-50-85f2dd7c40bd>", line 1, in <module>
    df_out = con.select_ipc_gpu("select * from int_table")

  File "/home/rzwitch/pymapd/pymapd/connection.py", line 278, in select_ipc_gpu
    df = _parse_tdf_gpu(tdf)

  File "/home/rzwitch/pymapd/pymapd/_parsers.py", line 190, in _parse_tdf_gpu
    dptr = ipch.open(ctx)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1127, in open
    return fn(context)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1112, in open_direct
    return self._impl.open(context)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1009, in open
    mem = context.open_ipc_handle(self.handle, self.offset + self.size)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 794, in open_ipc_handle
    driver.cuIpcOpenMemHandle(byref(dptr), handle, flags)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 290, in safe_cuda_api_call
    self._check_error(fname, retcode)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 325, in _check_error
    raise CudaAPIError(retcode, msg)

CudaAPIError: Call to call_cuIpcOpenMemHandle results in CUDA_ERROR_ALREADY_MAPPED

Registering it here, planning on trying to fix this myself.

[randyzwitch]

Calling ipch.close() in _parse_tdf_gpu seems to fix the issue above, at the expense of generating a separate error:

gdf.head().to_pandas()
ERROR:Call to cuMemcpyDtoD results in CUDA_ERROR_INVALID_VALUE
Traceback (most recent call last):

  File "<ipython-input-13-176dd1d28490>", line 1, in <module>
    gdf.head().to_pandas()

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/cudf/dataframe/dataframe.py", line 2058, in to_pandas
    out[c] = x.to_pandas(index=index)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/cudf/dataframe/series.py", line 670, in to_pandas
    s = self._column.to_pandas(index=index)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/cudf/dataframe/numerical.py", line 199, in to_pandas
    return pd.Series(self.to_array(fillna='pandas'), index=index)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/cudf/dataframe/column.py", line 309, in to_array
    return self.to_dense_buffer(fillna=fillna).to_array()

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/cudf/dataframe/column.py", line 504, in to_dense_buffer
    return self.data.copy()

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/cudf/dataframe/buffer.py", line 184, in copy
    return Buffer(mem=cudautils.copy_array(self.mem),

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/cudf/utils/cudautils.py", line 128, in copy_array
    out.copy_to_device(arr)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context
    return fn(*args, **kws)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 188, in copy_to_device
    _driver.device_to_device(self, ary, self.alloc_size, stream=stream)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1871, in device_to_device
    fn(device_pointer(dst), device_pointer(src), size, *varargs)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 290, in safe_cuda_api_call
    self._check_error(fname, retcode)

  File "/home/rzwitch/miniconda3/envs/cudf06/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 325, in _check_error
    raise CudaAPIError(retcode, msg)

CudaAPIError: Call to cuMemcpyDtoD results in CUDA_ERROR_INVALID_VALUE

The error occurs after ipch.close(), then trying to call gdf.to_pandas() Not sure if its related, but it's the same CUDA_ERROR_INVALID_VALUE error https://github.com/rapidsai/cudf/issues/1205

[randyzwitch]

Seems to be resolved on my machine now, not sure if this was a CUDA install mismatch on my machine or what. Closing for now, will re-open if observed again