Test for Windows and Python 3.8

[xhochy]

[dragoljub]

I'm seeing the same build issues on Windows 10 with Python 3.8 as your CI tests. It might be related to the BOOST library. I was able to build with Python 3.7 after installing/building boost and setting the %BOOST_ROOT% env variable. Unfortunately the same did not work for Python 3.8.

Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import turbodbc
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-6cf0b4450009> in <module>
----> 1 import turbodbc

d:\venv\attd_py38\lib\site-packages\turbodbc\__init__.py in <module>
      2
      3 from .api_constants import apilevel, threadsafety, paramstyle
----> 4 from .connect import connect
      5 from .constructors import Date, Time, Timestamp
      6 from .exceptions import Error, InterfaceError, DatabaseError, ParameterError

d:\venv\attd_py38\lib\site-packages\turbodbc\connect.py in <module>
      4 import six
      5
----> 6 from turbodbc_intern import connect as intern_connect
      7
      8 from .exceptions import translate_exceptions, ParameterError

ImportError: DLL load failed while importing turbodbc_intern: A dynamic link library (DLL) initialization routine failed.

[dragoljub]

I also saw this in my build log not sure if this matters;

D:\VENV\PY38\lib\site-packages\wheel\pep425tags.py:80: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  if get_flag('Py_DEBUG',

[keitherskine]

For what it's worth, I too am having difficulty getting Python 3.8 to play nice with turbodbc on Windows 10. Using the VS2019 compiler with Boost 1.72, turbodbc appears to pip install successfully, but then I get the same ImportError error when I import turbodbc:

Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import turbodbc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\keith\Downloads\pyvenvs\py38\lib\site-packages\turbodbc\__init__.py", line 4, in <module>
    from .connect import connect
  File "C:\Users\keith\Downloads\pyvenvs\py38\lib\site-packages\turbodbc\connect.py", line 6, in <module>
    from turbodbc_intern import connect as intern_connect
ImportError: DLL load failed while importing turbodbc_intern: A dynamic link library (DLL) initialization routine failed.
>>>

The only significant difference I can think of between Python 3.7 and 3.8 is that DLLs are no longer searched for in the PATH (see here for more info), but I'm not sure if that's even a factor here.

Then again, perhaps a newer of pybind11 is needed. There have been a couple of apparently minor fixes for Python 3.8 recently (see here), but maybe those fixes are relevant in this case. Difficult to see how they would be specific to Windows though.

[jogolemo]

Hey everyone. I'm running into this same problem. I installed Boost, set the BOOST_ROOT, was able to successfully pip install turbodbc in my venv, but when I try to import turbodbc or "from turbodbc import connect" I receive the same error (below):

(.venv) C:\path\to\repository>pip install -U turbodbc Collecting turbodbc Using cached turbodbc-4.0.0.tar.gz (77 kB) Requirement already satisfied, skipping upgrade: pybind11>=2.2.0 in c:\path\to.venv\lib\site-packages (from turbodbc) (2.5.0) Using legacy setup.py install for turbodbc, since package 'wheel' is not installed. Installing collected packages: turbodbc Running setup.py install for turbodbc ... done Successfully installed turbodbc-4.0.0`

(.venv) C:\path\to\repository>python Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import turbodbc Traceback (most recent call last): File "", line 1, in File "C:\path\to.venv\lib\site-packages\turbodbc_init_.py", line 2, in from .connect import connect File "C:\path\to.venv\lib\site-packages\turbodbc\connect.py", line 1, in from turbodbc_intern import connect as intern_connect ImportError: DLL load failed while importing turbodbc_intern: A dynamic link library (DLL) initialization routine failed.

I am using Python 3.8.0. Anyone have a workaround / fix?

[xhochy]

I am using Python 3.8.0. Anyone have a workaround / fix?

No, otherwise this PR wouldn't still be open.

[keitherskine]

I'm pretty certain now this issue is due to the change in behavior for Python 3.8. On Windows, there are environment variables for each user and "system" environment variables for everybody. Before Python 3.8, both system and user PATHs were examined to find DLLs for extension modules. Now in Python 3.8, only the system PATH is examined. However, if you have Python installed only for you (as opposed to for everybody on your PC), then the Python library paths are in your user environment variables, not the system environment variables.

Python itself has a DLL that is located in the top-level installation directory:

C:\Program Files\PythonXX\pythonXX.dll                                <-- everybody
C:\Users\<user>\AppData\Local\Programs\Python\PythonXX\pythonXX.dll   <-- user only

Hence, if an extension module needs the Python DLL, in Python 3.8, it will not be found in user-only Python installations. This appears to be what's happening on my PC. For some reason, turbodbc needs pythonXX.dll, yet after pip install turbdobc, the compiled pyd file in Lib\site-packages\ does not have access to a working copy:

(I used Dependencies to get this)

In Python 3.7, the pyd file has the Python DLL path found in my user environment variables: (Yes, that's an Amazon Python DLL, which is odd, but kind of a separate issue).

Unfortunately, it doesn't seem to be possible to install a local turbodbc repo on my PC with python setup.py install, so it's difficult to debug this and come up with a solution. It looks like a call to os.add_dll_directory() may have to be added somewhere, probably in setup.py, or perhaps the make files will have to be updated. For example, the following call should include the Python installation directory, even within venvs:

os.add_dll_directory(sys.base_prefix)  # note, not sys.prefix

[xhochy]

Thanks @keitherskine for looking into this so detailed. We should not be linking at all with python38.dll as far as my understanding goes (you should only link against it if you need the interpreter which we don't). I can look at fixing the build accordingly on Monday.

[xhochy]

The linkage against python38.dll is not actually the problem. There must be something specific about turbodbc_intern.cp38-win_amd64.pyd as importing turnodbc_numpy_support works for me.

[wanstr]

Hi everyone, just checking, is there any update on this issue? Would love to use with python 3.8.

[adament]

@xhochy If I want to add tests for #316 as requested by @fjetter testing the module on Windows and Python >= 3.8, should I just merge this pull request into mine? It looks to me like the CI infrastructure has changed since this pull request, but I am unfamiliar with it.