pythran
pythran copied to clipboard
serge-sans-paille
numpy.dot compiling fails (python 3.6, IPython 6.1.0, Pythran 0.8.5, win10)
Hi, First of all, thanks for this awesome package.
I have an issue and a question:
1 - I am trying to compile a function with numpy.dot product and the compiling fails (but if I remove the part with the dot product and leave the rest it works, aleternatively, if I only leave the dot product it fails):
eg:
%load_ext pythran.magic
%%pythran
#pythran export fun(float[][],float[][])
import numpy as np
def fun(A,B):
return np.dot(B.A)
Any thoughts?
2 -My question is about the compiling. In IPyhton, I often create functions in seperate files that I call like modules, as long as they are in the same directory as my working directory. How can I call such a file compiled with Pythran?
Thanks a lot
Hi Akeraa,
1 - can you provide me with the full compiler error? I see you're running on windows which is not the most tested setup....
2 - you need to compile this file with pythran (pythran my_module.py ) and it will create a native version of the module in the same directory. The Python import mechanism prefers this native module over the interpreted one.
Hi Serge-Sans-Paille, Thanks a lot for your reply. Here is the compile error: Compile_error.txt
For the question (2), it is what I did, I just now realised the native module was saved in the "Scripts" folder of Anaconda and not in the directory where my original ".py" file was. I guess it goes where pythran is called from.
Thanks, it helps a lot ! Do you have openblas installed somewhere ?
Hum, not that I am aware of. Should I install it?
EDIT: Only v0.2.19 has windows binary packages.
EDIT2: I did conda install -c menpo openblas, which installed openblas-0.2.13
I have 'cblast,h' included in 'C:\ProgramData\Anaconda3\Library\include'
I still get the exact same error
not yet, I'll check how numpy handles that to make deployment easier. Keep in touch!
What happens if you change blas=blas into blas=openblas in your pythran-win32.cfg ?
Thanks, I get quite a similar output but with some additional lines at the top: Compile_error_openblast.txt FYI, my pythran-win32.cfg is now: [compiler] defines= undefs= include_dirs= libs= library_dirs= cflags=/std:c++14 ldflags= blas=openblas CC= CXX=
Interestingly I tried to run my code on Google collab (colab.research.google.com/) on an online kernel, online ipynb. I don't know if it helps:
`%%pythran
#pythran export fun(float[][],float[][])
import numpy as np
def fun(A,B):
return np.add(A,B)`
seems to work. But
`%%pythran
#pythran export fun(float[][],float[][])
import numpy as np
def fun(A,B):
return np.dot(A,B)`
gives:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-9-c7d2bc09eeeb> in <module>()
----> 1 get_ipython().run_cell_magic('pythran', '', '#pythran export fun(float[][],float[][]) \nimport numpy as np\ndef fun(A,B):\n return np.dot(A,B)')
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2115 magic_arg_s = self.var_expand(line, stack_depth)
2116 with self.builtin_trap:
-> 2117 result = fn(magic_arg_s, cell)
2118 return result
2119
<decorator-gen-606> in pythran(self, line, cell)
/usr/local/lib/python3.6/dist-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
186 # but it's overkill for just that one bit of state.
187 def magic_deco(arg):
--> 188 call = lambda f, *a, **k: f(*a, **k)
189
190 if callable(arg):
/usr/local/lib/python3.6/dist-packages/pythran/magic.py in pythran(self, line, cell)
70 module_name = "pythranized_" + m.hexdigest()
71 module_path = pythran.compile_pythrancode(module_name, cell, **kwargs)
---> 72 module = imp.load_dynamic(module_name, module_path)
73 self._import_all(module)
74
/usr/lib/python3.6/imp.py in load_dynamic(name, path, file)
341 spec = importlib.machinery.ModuleSpec(
342 name=name, loader=loader, origin=path)
--> 343 return _load(spec)
344
345 else:
/usr/lib/python3.6/importlib/_bootstrap.py in _load(spec)
/usr/lib/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)
/usr/lib/python3.6/importlib/_bootstrap.py in module_from_spec(spec)
/usr/lib/python3.6/importlib/_bootstrap_external.py in create_module(self, spec)
/usr/lib/python3.6/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)
ImportError: /content/pythranized_c229e4990191302dc1d407b7993590fe.so: undefined symbol: _Z11cblas_dgemm11CBLAS_ORDER15CBLAS_TRANSPOSES0_iiidPKdiS2_idPdi
---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
That's still the same issue: Pythran needs to find a working version of blas on the system. The good point is that there's one in the official numpy wheel, I'll try to find a way to have Pythran use it. Keep in touch!
If you installed numpy from the wheels, there should be an openblas file somewhere in the numpy source. What's its path?
Sorry, I am not sur what you mean. I installed everything with Anaconda. I don't have any openblast file in my anaconda folder by default. All I could find are:
That's still the same issue: Pythran needs to find a working version of blas on the system. The good point is that there's one in the official numpy wheel, I'll try to find a way to have Pythran use it. Keep in touch!
Any update on this?
Getting a very similar error for x * y or np.multiply(x, y):
%%pythran
#pythran export pg_det(float[], float[])
import numpy as np
def pg_det(x, y):
x = np.asfarray(x)
y = np.asfarray(y)
return np.multiply(x, y) # same for "x * y", but works with "return 1"
I'm on Python 3.6, IPython 6.5.0, Pythran 0.8.6, Win10 64bit, and the dprod-example from the docs compiles.
@serge-sans-paille
If you installed numpy from the wheels, there should be an openblas file somewhere in the numpy source. What's its path?
I installed from wheel and get the following for np.show_config():
mkl_info:
libraries = ['mkl_rt']
library_dirs = ['C:/ProgramData/Miniconda3/envs/pythran\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Miniconda3/envs/pythran\\Library\\include']
blas_mkl_info:
libraries = ['mkl_rt']
library_dirs = ['C:/ProgramData/Miniconda3/envs/pythran\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Miniconda3/envs/pythran\\Library\\include']
blas_opt_info:
libraries = ['mkl_rt']
library_dirs = ['C:/ProgramData/Miniconda3/envs/pythran\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Miniconda3/envs/pythran\\Library\\include']
lapack_mkl_info:
libraries = ['mkl_rt']
library_dirs = ['C:/ProgramData/Miniconda3/envs/pythran\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Miniconda3/envs/pythran\\Library\\include']
lapack_opt_info:
libraries = ['mkl_rt']
library_dirs = ['C:/ProgramData/Miniconda3/envs/pythran\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Miniconda3/envs/pythran\\Library\\include']
Side note: "pythran" is the name of my conda environment for testing pythran, which might be confusing in the above paths.
Can you list the content of C:/ProgramData/Miniconda3/envs/pythran\\Library\\lib and tell me if there is an openblas library there?
@serge-sans-paille
All the .lib files without the ones starting with Qt or qt:
freetype.lib
icudt.lib
icuin.lib
icuio.lib
icutest.lib
icutu.lib
icuuc.lib
jpeg.lib
libeay32.lib
libeay32_static.lib
libEGL.lib
libGLESv2.lib
libiomp5md.lib
libiompstubs5md.lib
libjpeg.lib
libpng.lib
libpng16.lib
libpng16_static.lib
libpng_static.lib
libsodium.lib
libsodium_static.lib
libzmq-mt-4_2_5.lib
libzmq-mt-s-4_2_5.lib
libzmq.lib
mkl_blacs_ilp64_dll.lib
mkl_blacs_lp64_dll.lib
mkl_cdft_core_dll.lib
mkl_core_dll.lib
mkl_intel_ilp64_dll.lib
mkl_intel_lp64_dll.lib
mkl_intel_thread_dll.lib
mkl_pgi_thread_dll.lib
mkl_rt.lib
mkl_scalapack_ilp64_dll.lib
mkl_scalapack_lp64_dll.lib
mkl_sequential_dll.lib
mkl_tbb_thread_dll.lib
sqlite3.lib
ssleay32.lib
ssleay32_static.lib
z.lib
zdll.lib
zlib.lib
zlibstatic.lib
So no openblas, but (as far as I can tell) it's in the mkl_ stuff.
Yeah, one needs to find where the blas functions are defined in those... any educated guess?
I'd say mkl_intel_ilp64_dll.lib or mkl_intel_lp64_dll.lib (guessing based on size and trying to find some strings in the lib). I found a description of the differences between ilp and lp here, not sure how useful:
https://software.intel.com/en-us/mkl-macos-developer-guide-using-the-ilp64-interface-vs-lp64-interface
#1296 introduces a solution to this: depend on the pythran-openblas package which ships a static version of openblas. Eventually this is going to solve your issue (and to be the default for windows)
For the people using Anaconda Python, the default installation for NumPy / SciPy / etc. is MKL backed BLAS/LAPACK. To use it with Pythran you need to setup a virtual environment (if you want OpenBLAS) like this:
conda create -n openblas pip # we create a separate environment called openblas, and install pip there too
conda activate openblas # switch over to the new environment
pip install numpy
pip install scipy
pip install pythran
pip install pythran-openblas
pip install (whatever it complains about you not having, pandas or any other function in your script)...
Then you should install Microsoft 2019 Build Tools which is now here: https://my.visualstudio.com/Downloads?q=visual%20studio%202019&wt.mc_id=o~msft~vscom~older-downloads once you get the downloader, you can select the "C++ Clang tools for Windows" and get a working Clang-12 version of clang-cl.exe which Pythran can build PYD modules with. You should run the "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvarsall.bat x64" or whatever your architecture is before running Pythran so it can find the compiler. I setup on my environment just a simple shortcut like this:
%windir%\System32\cmd.exe "/K" C:\Users\[my_user_name]\Anaconda3\Scripts\activate.bat openblas
But, I also modified the activate.bat with a line to call the vcvarsall.bat automatically like this:
@CALL "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
Then compiling with a simple: pythran script_name.py should work fine. Actually I put in instructions to get this working in most cases for both OpenBLAS and Intel MKL w/ OpenMP here... until it makes it into the manual: https://github.com/serge-sans-paille/pythran/blob/8158734b43c8327cd63031ec333866911753d43f/README.rst
Still MKL and Numpy dot throws an error: error LNK2001: unresolved external symbol cblas_dgemm ... which is strange as no other supported functions throw an error. OpenBLAS seems okay with numpy.dot, however.
