How to make it work on Windows?

[NosIreland]

Hi, I want to use this in my code but getting problems to make it work on windows with python 3.4 I've installed it with pip without a problem but whenever I use import libarchive it fails with: OSError: [WinError 126] The specified module could not be found This is coming from ffi.py from the code below:

libarchive_path = os.environ.get('LIBARCHIVE') or find_library('archive') 
libarchive = ctypes.cdll.LoadLibrary(libarchive_path)

I've never used ctypes before but if I understand correctly it is looking for external DLL. I found and installed http://gnuwin32.sourceforge.net/packages/libarchive.htm also I've added C:\Program Files (x86)\GnuWin32\bin to my %PATH% in environmental variables but it still cannot load the module. As it does not give me the name of the module, I'm not sure what module it is looking for. What am I missing?

[pombredanne]

Sorry that I dropped the ball on the related #8 and #16 tickets....

I have prebuilt Windows binaries for another project here: https://github.com/nexB/scancode-toolkit/tree/develop/src/extractcode/bin/win-32/bin

The corresponding source code is there: https://github.com/nexB/scancode-thirdparty-src With MinGW32 build instructions there: https://github.com/nexB/scancode-thirdparty-src/blob/master/libarchive/build.sh#L47

The way scancode loads the library is a tad more engaged since it is the same code for Win/Linux/mac. You can see it there: https://github.com/nexB/scancode-toolkit/blob/develop/src/extractcode/libarchive2.py#L64 Note that this is NOT using python-libarchive-c yet but a different/custom ctypes binding for now. Until I have time to fix things. At least it gives you access to a Win DLL and an example on how to load it correctly.

[NosIreland]

Thanks for replay I'll see if I can make it work :)

[pombredanne]

I also posted a reply at http://stackoverflow.com/questions/37165002/using-libarchive-in-python-on-windows I assume that this is you on both places. ;)

[Zitrax]

Any possibility/plan of getting a pip install setting it all up to work without any additional manual steps on Windows ?

[Changaco]

@Zitrax Yes, someone just has to step up and do the work. I'm busy and I don't use Windows so it probably won't be me (except perhaps if someone proposes to pay me to get it done).

[HatScripts]

I have prebuilt Windows binaries for another project here: https://github.com/nexB/scancode-toolkit/tree/develop/src/extractcode/bin/win-32/bin

404... 😢

[pombredanne]

@HatScripts https://github.com/nexB/scancode-toolkit/tree/9e04eb55cd4a049409b86dac0b76445d87474c3a/plugins/extractcode-libarchive-win_amd64/src/extractcode_libarchive/lib

[HatScripts]

@pombredanne Thanks for that.

I still can't figure out how to make it work. I assume I need the libarchive.dll file, so I've tried adding that to my PATH, and also setting the LIBARCHIVE environment variable to point to it, but I get this error upon trying to import libarchive:

Traceback (most recent call last):
  File "D:/IdeaProjects/Python/Downloader/libarchive_test.py", line 4, in <module>
    import libarchive
  File "...\AppData\Local\Programs\Python\Python36\lib\site-packages\libarchive\__init__.py", line 1, in <module>
    from .entry import ArchiveEntry
  File "...\AppData\Local\Programs\Python\Python36\lib\site-packages\libarchive\entry.py", line 6, in <module>
    from . import ffi
  File "...\AppData\Local\Programs\Python\Python36\lib\site-packages\libarchive\ffi.py", line 27, in <module>
    libarchive = ctypes.cdll.LoadLibrary(libarchive_path)
  File "...\AppData\Local\Programs\Python\Python36\lib\ctypes\__init__.py", line 426, in LoadLibrary
    return self._dlltype(name)
  File "...\AppData\Local\Programs\Python\Python36\lib\ctypes\__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application

What am I doing wrong?

[pombredanne]

It is kinda hard to figure out short of looking at your code. Is this public? And FWIW, the build I listed above needs ALL the DLLs, not just libarchive.dll.

[MartinFalatic]

From your development branch I see:

Windows 64-bit libraries: https://github.com/nexB/scancode-toolkit/tree/develop/plugins/extractcode-libarchive-win_amd64/src/extractcode_libarchive/lib

Windows 32-bit libraries: https://github.com/nexB/scancode-toolkit/tree/develop/plugins/extractcode-libarchive-win32/src/extractcode_libarchive/lib

~~Judging by the error, I would think @HatScripts needs the Win32 libraries.~~

Well, I get the same error and I'm using Windows 10 x64 to try this out, and I've tried both libraries (that are in their folders with all the related libraries)

[MartinFalatic]

Oh. @pombredanne, the full contents of the libarchive libraries those two folders are completely byte-identical, and both appear to actually be the Win32 builds of these DLLs. The setup.cfg files do differ, but that doesn't seem to have had the desired effect when you built them.

Looking at the other library folders, libmagic is similarly afflicted.

The 7z library is the only one with distinct 32-bit and 64-bit binaries present.

This is all as seen on the develop branch here:

https://github.com/nexB/scancode-toolkit/tree/develop/plugins

[MartinFalatic]

Well, almost three hours later and I finally built a version of the libarchive binary from scratch that loaded.... but doesn't work enough to pass more than a few tests. That's an utterly miserable experience for something that should be precompiled to begin with.

[bicobus]

@pombredanne Would it be possible for you to include you build scripts to this project's setup.py? So that whenever somebody do a pip install libarchive-c, the build system triggers if libarchive isn't already present on the host system.

This would help with using external build system like azure-pipelines or appveyor.

[ganego]

Here is a guide I wrote in some other issue for some other project regarding how to compile libarchive on Windows:

I will now explain how to compile libarchive.dll, so that one can install this python libarchive package.

• Get the x64 installer from https://www.msys2.org/ and run it.

• Go to the installation folder and run mingw64.exe.

• Run pacman -Syu, restart the shell and run pacman -Su.

• Install minGW-64 via pacman -S mingw-w64-x86_64-toolchain.

• Install minGW-64-cmake (not normal cmake, as this will lack minGW generators) via pacman -S mingw-w64-x86_64-cmake.

• Install libxml2 via pacman -S mingw-w64-x86_64-libxml2.

• Restart the shell.

• Download and extract the libarchive source: https://github.com/libarchive/libarchive/releases

• In the msys2 shell navigate to the extraced folder. You can access Windows paths via cd /c/folder/subfolder for c:\folder\subfolder.

• Create makefiles via cmake . -G "MinGW Makefiles". You might have to run this twice, as the first time you might get an error about sh.exe.

• Compile via mingw32-make.exe.

• libarchive.dll is now in <libarchive_source\bin\libarchive.dll.

• In order to use this .dll you also need other dlls from the <msys2>\mingw64\bin folder. Copy those .dlls to the libarchive.dll folder:

• libbz2-1.dll

• libexpat-1.dll

• libiconv-2.dll

• liblz4.dll

• liblzma-5.dll

• libnettle-7.dll

• libxml2-2.dll

• libzstd.dll

• zlib1.dll

I'm not sure you need to do this to install python-libarchive-c, as this was from a post on a different project that needed libarchive in PATH for the installer Now that you got all those .dlls in one place, you have to make sure, the installer finds them when running pip:

• Bascially they have to be inside PATH, so you could copy them to C:\Windows\System32\, which would be the easiest way.

Alternatively, install python-libarchive-c and the put the .dll files into your program/script folder where you need libarchive:

• Run cmd and execute set Path=%PATH%;<libarchive_source>\bin (replace the <...> part). Do not add any quotation marks, even if the path has spaces. This just adds to the PATH for this cmd window, not the system PATH.
• Run pip install python-libarchive-c.
• Copy all .dll files into the folder where your .py file is and add this to the top of your script (above the libarchive import):
  os.environ["LIBARCHIVE"] = os.path.dirname(os.path.abspath(__file__), "libarchive.dll"). In case you want to distribute the script you should probably choose this way.

[pombredanne]

All and @HatScripts and @MartinFalatic in particular: sorry for missing some of the discussions and your messages in this tickets! @ganego your script rocks!

FWIW, the way I did build the natives has always been there on my side https://github.com/nexB/scancode-thirdparty-src/blob/master/libarchive/build.sh#L57 and there are build bits for each in https://github.com/nexB/scancode-thirdparty-src

This has always been painfully manual and it would be awesome to have these bakes in a setup.py of sorts : this is feasible though it will hard

[pombredanne]

Also I now have separate wheels built for scancode in https://github.com/nexB/scancode-toolkit/tree/develop/plugins-builtin that was contributed to help supporting Debian packaging and similar. For instance this https://github.com/nexB/scancode-toolkit/tree/develop/plugins-builtin/extractcode_libarchive-win_amd64/src/extractcode_libarchive/lib

We could very much extract that entirely of ScanCode and make it a reusable wheel(s) for everyone to reuse. From experience, that's not trivial to get right and is serious and complex work to get these kind of builds working.

If someone is a student willing to participate to the Google Summer of Code under scancode/aboutcode I would be willing to sponsor and mentor that FWIW!

[pombredanne]

@MartinFalatic re:

Oh. @pombredanne, the full contents of the libarchive libraries those two folders are completely byte-identical, and both appear to actually be the Win32 builds of these DLLs.

yes that was... my bad there. I am tackling the rebuild on all OS now.

[GadgetSteve]

I did find that downloading the appropriate zip file from https://github.com/libarchive/libarchive/releases and putting the archive.dll file into a directory pointed to by the LIBARCHIVE environmental variable worked, at least for my quick tests. Would it be worth considering releasing windows wheels with that dll from the libarchive releases in and with the library installed path (from the file value within ffi.py) considered as a possible library load point. I think that this would be considerably easier and more maintainable.

[Changaco]

I think it would be cleaner to make a separate package for people who want to install a copy of libarchive via pip. It could be named libarchive-binary or whatever. libarchive-c could be modified to use it automatically when it's installed and recommend installing it when no suitable libarchive is found.

[GadgetSteve]

@Changaco I quite like the idea - the only downside would be that the libarchive-binary would do nothing on its own - the upside would be that potentially it could contain the various different builds of libarchive (dll/etc) to support platforms other than Windows that may not have it installed or even when the system libarchive is outdated but cannot be updated.

[ganego]

I used libarchive-c because standard libarchive was not able to properly load files with non-ASCII names. At least this is the comment I have in my source code.
Does it work by now?
EDIT: Probably was about the pip archive one which is highly outdated. A quick test with the recent archive.dll from the github seems to work.