mkl_fft
mkl_fft copied to clipboard
IntelPython
ValueError: Internal error occurred: b'Intel MKL DFTI ERROR: Inconsistent configuration parameters' for certain shapes in _numpy_fft.rfft
For certain large 1d rffts, I get the following error. Tested this in anaconda python 3.7 and 3.8 on linux. If the long/fft axis is 1 instead of 0, I don't get this error.
ValueError: Internal error occurred: b'Intel MKL DFTI ERROR: Inconsistent configuration parameters'
This example generates the error below
import numpy as np
from mkl_fft._numpy_fft import rfft
X = np.random.randn(18134053, 1)
rfft(X, axis=0) # ok
print(18134052, 1)
X = np.random.randn(18134053, 16)
rfft(X, axis=0) # generates error
print(18134052, 16)
18134052 1
Traceback (most recent call last):
File "mkl_test.py", line 10, in <module>
rfft(X, axis=0)
File "/home/jesse/anaconda3/envs/mkl_test/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 414, in rfft
(x,), {'n': n, 'axis': axis})
File "/home/jesse/anaconda3/envs/mkl_test/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 103, in trycall
raise ve
File "/home/jesse/anaconda3/envs/mkl_test/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 98, in trycall
res = func(*args, **kwrds)
File "mkl_fft/_pydfti.pyx", line 792, in mkl_fft._pydfti.rfft_numpy
File "mkl_fft/_pydfti.pyx", line 701, in mkl_fft._pydfti._rc_fft1d_impl
ValueError: Internal error occurred: b'Intel MKL DFTI ERROR: Inconsistent configuration parameters'
This is in a basic conda environment with only numpy added.
$ conda list
# packages in environment at /home/jesse/anaconda3/envs/mkl_test:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
blas 1.0 mkl
ca-certificates 2021.1.19 h06a4308_0
certifi 2020.12.5 py37h06a4308_0
intel-openmp 2020.2 254
ld_impl_linux-64 2.33.1 h53a641e_7
libedit 3.1.20191231 h14c3975_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libstdcxx-ng 9.1.0 hdf63c60_0
mkl 2020.2 256
mkl-service 2.3.0 py37he8ac12f_0
mkl_fft 1.3.0 py37h54f3939_0
mkl_random 1.1.1 py37h0573a6f_0
ncurses 6.2 he6710b0_1
numpy 1.19.2 py37h54aff64_0
numpy-base 1.19.2 py37hfa32c7d_0
openssl 1.1.1j h27cfd23_0
pip 21.0.1 py37h06a4308_0
python 3.7.10 hdb3f193_0
readline 8.1 h27cfd23_0
setuptools 52.0.0 py37h06a4308_0
six 1.15.0 py37h06a4308_0
sqlite 3.33.0 h62c20be_0
tk 8.6.10 hbc83047_0
wheel 0.36.2 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
and
mkl.get_version()
# {'MajorVersion': 2020, 'MinorVersion': 0, 'UpdateVersion': 2, 'ProductStatus': b'Product',
# 'Build': b'20200624', 'Processor': b'Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors',
# 'Platform': b'Intel(R) 64 architecture'}
Possibly similar to #24.
cc @hlillemark
No errors with fft along axis 1
import numpy as np
from mkl_fft._numpy_fft import rfft
X = np.random.randn(1, 18134053)
rfft(X, axis=1)
print(1, 18134052)
X = np.random.randn(16, 18134053)
rfft(X, axis=1)
print(16, 18134052)
The same error happens for irfft, fft, ifft in _numpy_fft with this shape.
Thank you for the report @JesseLivezey.
This must be a recently introduced change.
In [1]: import numpy as np
In [2]: from mkl_fft._numpy_fft import rfft
In [3]: import mkl_fft
In [4]: X = np.random.randn(1, 18134053)
In [5]: Y = mkl_fft.rfft_numpy(X)
In [6]: Y.shape
Out[6]: (1, 9067027)
In [7]: Y = mkl_fft.rfft_numpy(X, axis=0)
In [8]: Y.shape
Out[8]: (1, 18134053)
In [9]: mkl_fft.__version__
Out[9]: '1.2.0'
Also, just having created an environment from conda-forge:
conda create -n cf_mkl_fft_issue_57 python ipython mkl numpy mkl_fft mkl-service
I can not reproduce the issue:
Python 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import numpy as np, mkl_fft, mkl
In [2]: X = np.random.randn(1, 18134053)
In [3]: Y = mkl_fft.rfft_numpy(X)
In [4]: Y0 = mkl_fft.rfft_numpy(X, axis=0)
In [5]: mkl.get_version_string()
Out[5]: 'Intel(R) Math Kernel Library Version 2020.0.4 Product Build 20200917 for Intel(R) 64 architecture applications'
I also can not reproduce the issue using the latest oneMKL version 2020.1, and the development version 2020.2 to be released soon.
@oleksandr-pavlyk I only get errors when the first dim is the long/fft dimension and the second is >1
X = np.random.randn(18134053, 2) # for me, anything >=2 for the second dim causes an error
In your tests, it looks like the second dimension is long, but you're taking the fft over the first.
Thank you @JesseLivezey I have reproduced the issue. It appears like MKL raises this exception when the dimension is greater than 2^24.
In [31]: Y0 = mkl_fft.rfft_numpy(X[slice(0, 2**24, 1), :], axis=0)
MKL_VERBOSE FFT(drfo16777216:2:2*2:1:1,bScale:5.96046e-08,tLim:6,unaligned_input,unaligned_output,desc:0x56293beeb700) 221.30ms CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:6
In [32]: Y0 = mkl_fft.rfft_numpy(X[slice(0, 2**24 + 1, 1), :], axis=0)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-32-c91c2fa3cd09> in <module>()
----> 1 Y0 = mkl_fft.rfft_numpy(X[slice(0, 2**24 + 1, 1), :], axis=0)
mkl_fft/_pydfti.pyx in mkl_fft._pydfti.rfft_numpy()
mkl_fft/_pydfti.pyx in mkl_fft._pydfti._rc_fft1d_impl()
ValueError: Internal error occurred: b'Intel MKL DFTI ERROR: Inconsistent configuration parameters'
I reproduced the internal error using C programming language and reported the issue to MKL team.
@oleksandr-pavlyk Apologies for reopening this old issue, but was there a resolution to this problem? We think we might be seeing a very similar issue in our code which directly calls the mkl runtime, and it would be good to rule out if it's an error on our end. Is there a link to the issue reporting (if public)?
@ahnitz Once the issue was originally reported, I changed mkl_fft to do a better job at reporting the error that MKL itself is flagging. I also notified relevant MKL team about the problem.
I have just checked that nightly builds of MKL still flag the input generated from input In[32] in my excerpt from Mar 8 comment as problematic.
I would therefore think the problem has not found its resolution as yet.
I encounter a similar issue when using ifft2d. Is there any progress on this ticket?
