[WIN10+MSYS2mingw64+PyGObject] ImportError: DLL load failed while importing _gi

[maurocsound]

Description of the issue

I'm on Windows 10 64bit. I would like to 'freeze' a python script (which uses GTK, so PyGObject package) into an exe. I correctly installed MSYS2 and updated the "MSYS2 MINGW 64-bit" environment as per instructions. I believe I installed (with pacman packages of the msys2 repository) all the required dependencies for python, setuptools, pip, pygobject, etc... I installed pyinstaller with the following command:

$ pip install pyinstaller

Then I ran, in my home dir (where I have my python script named 'main.py'):

$ pyinstaller main.py

It seems to go all right... There are no errors from the pyinstaller script. Now I have 2 dir: build and dist. I go into the dist/main subdir and when I try to run the generated exe I have the following error and the script doesn't run:

$ ./main.exe
Traceback (most recent call last):
  File "main.py", line 1, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 531, in exec_module
  File "gi/__init__.py", line 40, in <module>
ImportError: DLL load failed while importing _gi: The specified procedure could not be found.
[23628] Failed to execute script main

I also tried with the current development version of pyinstaller but it's the same thing...

Context information (for bug reports)

• Output of pyinstaller --version: 4.2

• Version of Python: 3.8.7

• Platform: Windows 10 Home 64bit

• tried also the latest development version, but it's the same thing...

A minimal example program which shows the error

# main.py:
import gi

print("Hello!")

Stacktrace / full error message

Traceback (most recent call last):
  File "main.py", line 1, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 531, in exec_module
  File "gi/__init__.py", line 40, in <module>
ImportError: DLL load failed while importing _gi: The specified procedure could not be found.
[23628] Failed to execute script main

[rokm]

Works as expected here with MSYS2 MinGW 64-bit prompt, so this might be specific to your environment...

Can you verify which python executable you are using and where pyinstaller shim was installed? I.e., what's the output of which python and which pyinstaller?

Also, check the dependencies of dist/[name]/program/gi/_gi-cpython-38.dll with e.g., Dependency Walker. (On my system, it depends on libffi-7.dll, libgirepository-1.0-1.dll, libglib-2.0-0.dll, libgobject-2.0-0.dll in addition to kernel32.dll, msvcrt.dll and libpython3.8.dll). Check if those glib-related DLLs are collected in dist/[name], and verify that they are identical to the DLLs that are installed in your MinGW64 environment.

You might want to try manually copying those DLLs from your MinGW64 environment one by one and see if it fixes the issue (The specified procedure could not be found. seems to indicate an incompatible DLL version...).

[maurocsound]

$ which python
/mingw64/bin/python


$ which pyinstaller
/mingw64/bin/pyinstaller

As per your suggestion, I tried to open that _gi-cpython-38.dll with Dependency Walker, but then it shows an alert window with the following:

Errors were detected when processing "c:\msys64\home\[username]\dist\main\gi\_GI-CPYTHON-38.DLL". See the log window for details.

Then in the log window:

Error: At least one required implicit or forwarded dependency was not found. Error: At least one module has an unresolved import due to a missing export function in an implicitly dependent module. Warning: At least one delay-load dependency module was not found. Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module.

And in the tree window of Dependency Walker there is a big amount of dll entries with a "(?)" mark...

[rokm]

Yeah, that's normal if those DLLs are not in the PATH when Dependency Walker was launched. The idea is to get the list of first-order dependencies for further investigation.

But you could add the base program directory to PATH and try again - this might also help you directly identify the problematic DLL (API-MS-* DLLs will probably remain unresolved anyway, but those are unlikely the cause of the problem).

[maurocsound]

In my case the first level dependencies are: LIBFFI-7.DLL KERNEL32.DLL MSVCRT.DLL LIBGIREPOSITORY-1.0-1.DLL LIBGLIB-2.0-0.DLL LIBGOBJECT-2.0-0.DLL LIBPYTHON3.8.DLL

I checked and the above DLLs (apart from kernel32.dll and msvcrt.dll) are all in the dist dir...

The (?) entries are only related to API-MS-*, EXT-MS-*, HVSIFILETRUST.DLL and IESHIMS.DLL.

[maurocsound]

Notice that if I don't use the "import gi" in the main.py script, the generated exe runs without errors.

I made another build with:

pyinstaller --debug=all main.py

Then, when I try to run the generated exe in dist/main subdir I get a more verbose traceback:

# gi not found in PYZ
# code object from 'C:/msys64/home/[username]/dist/main/gi/__init__.pyc'
# gi._gi not found in PYZ
Traceback (most recent call last):
  File "main.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:/msys64/mingw64/lib/python3.8/site-packages/gi/__init__.py", line 40, in <module>
    from . import _gi
  File "<frozen importlib._bootstrap>", line 1042, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1101, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing _gi: The specified procedure could not be found.
[22908] Failed to execute script main

Then I noticed (by using pyi-archive_viewer) that there is no 'gi._gi' entry in the PYZ-00.pyz archive file (inside the build dir)... Maybe is this the issue?

[rokm]

Then I noticed (by using pyi-archive_viewer) that there is no 'gi._gi' entry in the PYZ-00.pyz archive file (inside the build dir)... Maybe is this the issue?

No, the gi._gi is an extension module, and is therefore included as DLL outside the PYZ (gi/_gi-cpython-38.dll).

[maurocsound]

If I delete that dll the traceback error is slightly different:

# gi not found in PYZ
# code object from 'C:/msys64/home/[username]/dist/main/gi/__init__.pyc'
# gi._gi not found in PYZ
Traceback (most recent call last):
  File "main.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:/msys64/mingw64/lib/python3.8/site-packages/gi/__init__.py", line 40, in <module>
    from . import _gi
ImportError: cannot import name '_gi' from partially initialized module 'gi' (most likely due to a circular import) (C:/msys64/home/[username]/dist/main/gi/__init__.pyc)
[10716] Failed to execute script main

What else can I do?

[rokm]

What else can I do?

You'll need to figure out which DLL is triggering the The specified procedure could not be found. error. Perhaps try copying LIBFFI-7.DLL, LIBGIREPOSITORY-1.0-1.DLL, LIBGLIB-2.0-0.DLL and LIBGOBJECT-2.0-0.DLL from C:\msys64\mingw64\bin to dist\[name] (or check if they are identical, e.g. via hash).

(And just to check, does import gi work if ran directly from python in your environment?)

[maurocsound]

(And just to check, does import gi work if ran directly from python in your environment?)

Yes, it works correctly if I do: $ python main.py

(or if I write import gi directly inside the python command line interpreter...)

It's just from the generated exe inside the dist dir that it doesn't work...

[maurocsound]

Ok, I overwrote those DLLs from the c:\msys64\mingw64\bin dir to the dist\main dir and now it works... But I don't understand why... Doesn't pyinstaller just copy the relevant DLLs from that dir?

[maurocsound]

I found the culprits are the following:

LIBGLIB-2.0-0.DLL LIBGOBJECT-2.0-0.DLL

After each pyinstaller [prog_name].py execution I have to overwrite those 2 DLLs in the dist/[prog_name] dir to make the exe work. They have slight size differences between the 2 directories (c:\msys64\mingw64\bin and dist\[prog_name]). What can be the cause? Could it be a 'locale' problem? I'm from Italy so I use Windows 10 in Italian language...

[rokm]

Doesn't pyinstaller just copy the relevant DLLs from that dir?

It should, unless your PATH (or maybe LD_LIBRARY_PATH) also points to another directory that contains another version of those DLLs, and those end up being picked up instead. Which I suspect is happening in your case...

[maurocsound]

I see no LD_LIBRARY_PATH variable from the export command... Just PATH:

PATH="/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/[username]/dist/main"

[rokm]

How about if you open build/[prog_name]/COLLECT-00.toc and find the entry for libglib-2.0-0.dll? Where is it being taken from?

[maurocsound]

OMG! It's taken from 'C:/Program Files/Csound6_x64/bin/libglib-2.0-0.dll' (along with a bunch of other gobject libs)!!! I like to experiment with algorithmic sound generation, so I have Csound installed on my machine... I didn't even know it uses gobject libs... But if I'm in the msys2-mingw64 environment, why the Windows path (which includes the above csound bin dir) is taken instead of the msys2-mingw64 one?

EDIT: or maybe the windows path has priority over the mingw one...

[maurocsound]

The strange thing is that if I do a python main.py (from an msys2-mingw64 shell) it just works... So it looks to me that the python interpreter correctly takes the right DLLs from the right path... Why this is not the case for pyinstaller too?

[rokm]

Can you check sys.path in your python interpreter? I think that's the first place that PyInstaller uses for library resolution, and on my test system, the manual addition to base system's PATH seems to leak into sys.path in MinGW environment.

[maurocsound]

>>> sys.path
['', 'C:/Program Files/Csound6_x64/bin', 'C:/msys64/home/[username]', 'C:/msys64/mingw64/lib/python38.zip', 'C:/msys64/mingw64/lib/python3.8', 'C:/msys64/mingw64/lib/python3.8/lib-dynload', 'C:/msys64/mingw64/lib/python3.8/site-packages']

[rokm]

Can you check sys.path in your python interpreter? I think that's the first place that PyInstaller uses for library resolution, and on my test system, the manual addition to base system's PATH seems to leak into sys.path in MinGW environment.

Actually, the problem is that the CSound installer ends up adding its bin directory to PYTHONPATH (because it internally also uses python), and this affects all python installations on the system. And because PyInstaller uses sys.path as first choice when resolving DLLs, it ends up using CSound-bundled DLLs.

[rokm]

So the work-around here is to unset PYTHONPATH before running PyInstaller.

[maurocsound]

In PYTHONPATH I have:

PYTHONPATH='C:\Program Files\Csound6_x64\bin;C:\Program Files\Csound6_x64\bin;C:\Program Files\Csound6_x64\bin;'

I see only Csound related dirs there, so is it an environment variable set by Csound installer? And that's automatically added to sys.path in python?

[rokm]

Yes, PYTHONPATH is automatically added to sys.path, as per docs.

In your case, it's been set by Csound installer (presumably you installed/upgraded it three times).

Running export PYTHONPATH= in your MinGW shell should clear it for that session, though. If you run python interpreter afterwards, sys.path should not contain that Csound directory anymore. (And PyInstaller should pick up correct DLLs).

[maurocsound]

Ok, but then I fail to understand why with python main.py the interpreter loads the right DLLs... That's very confusing to me... Does this mean that the sys.path variable is NOT what it's internally used by the Python interpreter for path resolution of the modules? If that's the case, maybe pyinstaller should not use sys.path, but the real path used internally by the python interpreter so that it can get the correct path for DLLs even in these cases (at least to be "consistent" with what happens when I try to execute my script in the canonical way: python main.py).

I mean: the search path used in the case of python main.py and in the case of pyinstaller main.py ideally should be the same... But it looks actually they are different, right?

[rokm]

The difference is probably indeed coming from python interpreter not using sys.path / PYTHONPATH for shared library resolution.

Not sure why we search sys.path first, but it seems very deliberate according to that part of the code:

https://github.com/pyinstaller/pyinstaller/blob/c24117a98f4322de861f62019fb3c3c869a85245/PyInstaller/depend/bindepend.py#L69-L72

[maurocsound]

Maybe that was under the assumption that python would search first there... In that case the above comment would make sense... Unfortunately it looks like python internally doesn't use sys.path for shared library resolution... Maybe a bug in cpython code?

P.S.: I don't know if it's related but if I try to build with --onefile option, I get the following error:

$ ./main.exe
gi/_gi-cpython-38.dll could not be extracted!
fopen: No such file or directory

Without --onefile it works...

[rokm]

Maybe that was under the assumption that python would search first there... In that case the above comment would make sense... Unfortunately it looks like python internally doesn't use sys.path for shared library resolution... Maybe a bug in cpython code?

I suspect that python's behavior might have changed over time in this regard, and we haven't kept up.

P.S.: I don't know if it's related but if I try to build with --onefile option, I get the following error:

$ ./main.exe gi/_gi-cpython-38.dll could not be extracted! fopen: No such file or directory

Without --onefile it works...

This is probably unrelated, but I can reproduce it on my system as well, so it's definitely a bug. Will take a closer look to see what's going on.

[maurocsound]

P.S.: sorry, I noticed I forgot to thank you for your great support until now... Thanks! :wink:

[stormbeforesunsetbee]

OOT: Weirdly enough, now we have to implicitly add 'gi.repository.Gtk' to 'hidden_imports'. It wasn't like that in/before August 2021

[danyeaw]

This is an issue with PyGObject itself and is being tracked by: https://gitlab.gnome.org/GNOME/pygobject/-/issues/545

I would suggest we close this for PyInstaller.

[rokm]

@danyeaw Are you sure this is related to the issue you linked? If I recall correctly, the problem with this particular issue was that incompatible GObject DLLs were collected due to user having installed a 3rd party program that added its installation directory (which also contained GObject DLLs) to PYTHONPATH globally, and PyInstaller using sys.path as the first location to resolve the DLL paths.