mkl_fft icon indicating copy to clipboard operation
mkl_fft copied to clipboard

Limitation of: Data size of one of transform dimensions exceeds 2^31 - 1 bytes

Open JesseLivezey opened this issue 3 years ago • 0 comments

Based on the error message, it seems like this is a known limitation, but I'm resampling data using ffts and get an error related to the size of the data. The data shape is: (17863680, 128) and the fft is being taken along axis=0.

I'm guessing this might be related to an index being int32, and if it is possible to get rid of this limitation, that would be helpful.

  File "/home/jlivezey/process_nwb/process_nwb/fft.py", line 22, in rfft
    return mklrfft(*args, **kwargs)
  File "/clusterfs/bebb/users/jlivezey/anaconda3/envs/nsds_nwb/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 414, in rfft
    (x,), {'n': n, 'axis': axis})
  File "/clusterfs/bebb/users/jlivezey/anaconda3/envs/nsds_nwb/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 103, in trycall
    raise ve
  File "/clusterfs/bebb/users/jlivezey/anaconda3/envs/nsds_nwb/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 98, in trycall
    res = func(*args, **kwrds)
  File "mkl_fft/_pydfti.pyx", line 792, in mkl_fft._pydfti.rfft_numpy
  File "mkl_fft/_pydfti.pyx", line 701, in mkl_fft._pydfti._rc_fft1d_impl
ValueError: Internal error occurred: b'Intel MKL DFTI ERROR: Data size of one of transform dimensions exceeds 2^31 - 1 bytes'

JesseLivezey avatar Aug 01 '21 19:08 JesseLivezey