with dpctl.device_context fails

[reazulhoque]

The following program:

import dpctl
import numpy as np
import dpnp


with dpctl.device_context("opencl:gpu:0"):
    a = dpnp.arange(1021, dtype=np.int64)

Produces the following error:

DPNP: Module NumPy found. Please load DPNP module before NumPy.

  warnings.warn("\nDPNP: Module NumPy found. Please load DPNP module before NumPy.\n")
Running on: Intel(R) UHD Graphics [0x9bca]
DPCtrl SYCL queue used
SYCL kernels link time: 2.79e-07 (sec.)
Math backend version: Intel(R) oneAPI Math Kernel Library Version 2021.2-Product Build 20210312 for Intel(R) 64 architecture applications

terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  Native API failed. Native API returns: -33 (CL_INVALID_DEVICE) -33 (CL_INVALID_DEVICE)
Aborted (core dumped)

Versions:

dpnp 0.6.2 py38h86f26c8_18 dpctl 0.8.0a0

Please confirm if you can reproduce this.

[oleksandr-pavlyk]

Can you run the example with SYCL_PI_TRACE=-1, and if possible, provide a backtrace.

[reazulhoque]

@oleksandr-pavlyk SYCL_PI_TRACE.txt

[reazulhoque]

If we change the SYCL filter string from opencl:gpu:0 to level_zero:gpu:0 this error goes away. I suspect the following with opencl:gpu:0:

1. During allocation of the array the opencl:gpu:0 queue is used.
2. The deallocation happens outside the with context region and the free() is submitted to the default queue (which I suspect is level_zero).

I think the queue should be cached if not already being that way by dpnp and during the free() it should be submitted to the right queue.

[oleksandr-pavlyk]

Yes, @reazulhoque analysis is accurate:

In [1]: import dpctl, dpnp
/localdisk/work/opavlyk/repos/dpnp/dpnp/__init__.py:31: UserWarning:
DPNP: Module NumPy found. Please load DPNP module before NumPy.

  warnings.warn("\nDPNP: Module NumPy found. Please load DPNP module before NumPy.\n")
Running on: Intel(R) UHD Graphics [0x9bca]
DPCtrl SYCL queue used
SYCL kernels link time: 2.67e-07 (sec.)
Math backend version: Intel(R) oneAPI Math Kernel Library Version 2021.2-Product Build 20210312 for Intel(R) 64 architecture applications


In [2]:  with dpctl.device_context("opencl:gpu:0"):
   ...:     a = dpnp.arange(1021)
   ...:

In [3]: del a  # free is called on the current queue, but should use the queue the memory was allocated with
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  Native API failed. Native API returns: -33 (CL_INVALID_DEVICE) -33 (CL_INVALID_DEVICE)
Aborted (core dumped)

[PokhodenkoSA]

I see the same behavior on dpnp 0.7.1

[Alexander-Makaryev]

@PokhodenkoSA As we all (and you too :) ) decided, we are going to use dpctl.tensor.usm_ndarray as a data container and deprecate dpnp.dparray. So, this issue should be resolved with memory management in usm_array on dpctl side. dparray is still used as default container In dpnp version 0.7.1, but work with dpctl.tensor.usm_ndarray is available with DPNP_OUTPUT_DPCTL=1 environment variable. We are working on extending of it support and we are going to enable it as default container in future.

[PokhodenkoSA]

dparray is still used as default container In dpnp version 0.7.1, but work with dpctl.tensor.usm_ndarray is available with DPNP_OUTPUT_DPCTL=1 environment variable

Thanks for the explanation.

[PokhodenkoSA]

So the issue should go to dpctl.

@oleksandr-pavlyk Could you check please this scenario in dpctl with dpctl.tensor.usm_ndarray?