IndexError: vector::_M_range_check

[keisukefukuda]

When using pynvvl, the following error occurs repeatedly, although the exception is ignored and the execution can continue successfully. I guess it's more like CuPy error, so please close this issue if it's inappropriate to report here.

Thanks!

Exception ignored in: 'cupy.cuda.memory.PooledMemory.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/memory.pyx", line 544, in cupy.cuda.memory.PooledMemory.free
  File "cupy/cuda/memory.pyx", line 816, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 830, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 679, in cupy.cuda.memory.SingleDeviceMemoryPool._remove_from_free_list
IndexError: vector::_M_range_check
Traceback (most recent call last):
  File "cupy/cuda/memory.pyx", line 544, in cupy.cuda.memory.PooledMemory.free
  File "cupy/cuda/memory.pyx", line 816, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 835, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 679, in cupy.cuda.memory.SingleDeviceMemoryPool._remove_from_free_list
IndexError: vector::_M_range_check
Exception ignored in: 'cupy.cuda.memory.PooledMemory.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/memory.pyx", line 544, in cupy.cuda.memory.PooledMemory.free
  File "cupy/cuda/memory.pyx", line 816, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 835, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 679, in cupy.cuda.memory.SingleDeviceMemoryPool._remove_from_free_list
IndexError: vector::_M_range_check
Traceback (most recent call last):
  File "cupy/cuda/memory.pyx", line 544, in cupy.cuda.memory.PooledMemory.free
  File "cupy/cuda/memory.pyx", line 816, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 835, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 679, in cupy.cuda.memory.SingleDeviceMemoryPool._remove_from_free_list
IndexError: vector::_M_range_check
Exception ignored in: 'cupy.cuda.memory.PooledMemory.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/memory.pyx", line 544, in cupy.cuda.memory.PooledMemory.free
  File "cupy/cuda/memory.pyx", line 816, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 835, in cupy.cuda.memory.SingleDeviceMemoryPool.free
  File "cupy/cuda/memory.pyx", line 679, in cupy.cuda.memory.SingleDeviceMemoryPool._remove_from_free_list
IndexError: vector::_M_range_check

(loooooong errors)

[mitmul]

Thank you very much for reporting the error! I'll investigate it after de:code.

[keisukefukuda]

Thanks! I will come up with code to reproduce the issue.

[kmaehashi]

I think this issue will be solved by CuPy v4.1.0/v5.0.0b1. https://github.com/cupy/cupy/pull/1165

[dBeker]

I got this error with CuPy v5.0.0a1. I can confirm that CuPy v5.0.0b2 does not have this issue.