pyinstaller
unable to build bootloader
Description of the issue
Context information (for bug reports)
-
Output of
pyinstaller --version:(4.5.1) -
Version of Python: Python 3.9.7
-
Platform: OSX -> targeting arm64
-
How you installed Python: brew
-
Did you also try this on another platform? Does it work there? tried in venv, no
-
try the latest development version, using the following command:
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
- follow all the instructions in our "If Things Go Wrong" Guide (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and
Make sure everything is packaged correctly
- [ x] start with clean installation
- [ x] use the latest development version
- [x ] Run your frozen program from a command window (shell) — instead of double-clicking on it
- [ ] Package your program in --onedir mode
- [ ] Package without UPX, say: use the option
--noupxor setupx=Falsein your .spec-file - [ ] Repackage you application in verbose/debug mode. For this, pass the option
--debugtopyi-makespecorpyinstalleror useEXE(..., debug=1, ...)in your .spec file.
A minimal example program which shows the error
I started by cloning the pyinstaller code, and followed the directions contained within the Dockerfile
(paste text here)
docker run -v "$PWD:/io" -t arm64-full-test
Stacktrace / full error message
(venv) {USER}@{MAC} pyinstaller % sudo docker build --build-arg BASE=dockcross/linux-arm64-full -t arm64-full-test ./bootloader
[+] Building 0.2s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.80kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/dockcross/linux-arm64-full:latest 0.0s
=> [1/8] FROM docker.io/dockcross/linux-arm64-full 0.0s
=> CACHED [2/8] WORKDIR /home/ 0.0s
=> CACHED [3/8] RUN curl -s https://zlib.net/zlib-1.2.11.tar.gz | tar xfz - 0.0s
=> CACHED [4/8] WORKDIR zlib-1.2.11 0.0s
=> CACHED [5/8] RUN ./configure --prefix="$(/buildroot/bin/aarch64-buildroot-linux-gnu-gcc -print-sysroot)" 0.0s
=> CACHED [6/8] RUN make 0.0s
=> CACHED [7/8] RUN make install 0.0s
=> CACHED [8/8] WORKDIR /io 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:32f9198c79ea3f9372e696b660f4011257ffa9b8b52ee2fd98771b87251c0a1b 0.0s
=> => naming to docker.io/library/arm64-full-test 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
(venv) {USER}@{MAC} pyinstaller % docker run -v "$PWD:/io" -t arm64-full-test
'all' finished successfully (0.002s)
'distclean' finished successfully (0.002s)
Setting top to : /io/bootloader
Setting out to : /io/bootloader/build
Python Version : 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
Checking for 'gcc' (C compiler) : /buildroot/bin/aarch64-buildroot-linux-gnu-gcc
Checking size of pointer : 8
Platform : Linux-64bit-arm detected based on compiler
Checking for compiler flags -m64 : no
Checking for linker flags -m64 : no
Checking for library dl : yes
Checking for library pthread : yes
Checking for library m : yes
Checking for library z : yes
Checking for library cmocka : not found
Checking for function unsetenv : yes
Checking for function mkdtemp : yes
Checking for function dirname : yes
Checking for function basename : yes
Checking for function strndup : yes
Checking for function strnlen : yes
Checking for compiler flags -Wl,--as-needed : yes
Checking for program 'strip' : /usr/bin/strip
Checking for program 'strip' : /usr/bin/strip
'configure' finished successfully (2.610s)
'make_all' finished successfully (0.027s)
Waf: Entering directory `/io/bootloader/build/debug'
[ 1/16] Compiling src/pyi_path.c
[ 2/16] Compiling src/pyi_exception_dialog.c
[ 3/16] Compiling src/pyi_apple_events.c
[ 4/16] Compiling src/pyi_win32_utils.c
[ 5/16] Compiling src/pyi_utils.c
[ 6/16] Compiling src/pyi_main.c
[ 7/16] Compiling src/pyi_archive.c
[ 8/16] Compiling src/pyi_splash.c
[ 9/16] Compiling src/pyi_global.c
[10/16] Compiling src/pyi_splashlib.c
[11/16] Compiling src/pyi_python.c
[12/16] Compiling src/pyi_pythonlib.c
[13/16] Compiling src/pyi_launch.c
[14/16] Compiling src/main.c
[15/16] Linking build/debug/run_d
[16/16] Processing build/debug/run_d
/usr/bin/strip: Unable to recognise the format of the input file `run_d'
Waf: Leaving directory `/io/bootloader/build/debug'
Build failed
-> task in 'run_d' failed with exit status 1 (run with -v to display more information)
Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs for more about what would use to solve the issue.
I just tried the same thing on a fresh install of windows on a laptop. used the same version of python 3.9.7 and followed the directions closely yet still unable to build bootloader.
I think that's an unforeseen side effect of #6218. Can you try to specify STRIP environment variable with path to the strip executable from the toolchain that you use (it should probably be STRIP=/buildroot/bin/aarch64-buildroot-linux-gnu-strip).
I had to do something similar when I was putting together our osxcross Docker file:
https://github.com/pyinstaller/pyinstaller/blob/d53be0c8c9c961952ea1a08646d29fc920f12537/bootloader/Dockerfile.osxcross#L110
I think that's an unforeseen side effect of #6218. Can you try to specify
STRIPenvironment variable with path to thestripexecutable from the toolchain that you use (it should probably be STRIP=/buildroot/bin/aarch64-buildroot-linux-gnu-strip).I had to do something similar when I was putting together our
osxcrossDocker file:https://github.com/pyinstaller/pyinstaller/blob/d53be0c8c9c961952ea1a08646d29fc920f12537/bootloader/Dockerfile.osxcross#L110
Not sure how I would go about locating the "path to the strip executable from the toolchain that I use" I looked for the directory you mentioned, and looked inside of my venv bin directory but am not sure how to locate the strip executable.
Honestly I am not even sure what strip does... for clarity I am on a mac with BigSur 11.6 and using a venv trying to target arm64
Hmm, actually, I just noticed you're using PyInstaller 4.5.1, so your problem cannot be caused by #6218...
But the problem is very likely incompatibility between /usr/bin/strip and the output of the cross-compiler (so /buildroot/bin/aarch64-buildroot-linux-gnu-strip should be used).
What are you trying to do here, though? Cross-compile the bootloader for linux arm64?
for clarity I am on a mac with BigSur 11.6 and using a venv trying to target arm64
If you're trying to build macOS arm64 executable, you can use the existing universal2 bootloaders. But if you're running on x86_64 macOS, you won't be able to freeze your application with brew python, because brew python is single-arch. See the docs.
I am still very new to this, so I apologize if I am missing something that should be glaringly obvious. I am trying to create an executable file that can run on a linux (busybox) arm64 (ARMv8) that does not have python. I am currently on a MAC but have tried to do it on windows pc as well (both x86). I have another linux computer (x86) that I can try it on.
do I need to run pyinstaller on an arm system for it to work? I thought the purpose of building the bootloader was for cross compatability
do I need to run pyinstaller on an arm system for it to work? I thought the purpose of building the bootloader was for cross compatability
Even if you do cross-compile bootloader for target arch, you need to freeze your python script using environment running on that target arch, because PyInstaller needs to collect the python shared library and any other shared libraries / binary extensions from python packages that you use. So you cannot use PyInstaller to "cross-compile" your program for other arches (macOS itself being a bit of an exception there due to universal2, but even there it really depends).
do I need to run pyinstaller on an arm system for it to work? I thought the purpose of building the bootloader was for cross compatability
You're as well thinking of macOS's ARM and Linux's aarch64 (aka armv8) as unrelated - they have almost nothing in common. If I were you, I'd use your x86 Linux, install docker, install qemu, then use docker buildx to run an aarch64 docker image inside of x86. From inside that image you can install and run PyInstaller and everything it produces will be for ARM. That's a steep learning curve though...
Oh, and PyInstaller already ships ARM bootloaders for Linux so you shouldn't need to compile the bootloaders. In Linux arm64 is called aarch64 so if you download a wheel tagged aarch64 linux, that'll run on ARM. You still need an ARM Linux machine (or to virtualise one) though in order to run PyInstaller.
do I need to run pyinstaller on an arm system for it to work? I thought the purpose of building the bootloader was for cross compatability
You're as well thinking of macOS's ARM and Linux's aarch64 (aka armv8) as unrelated - they have almost nothing in common. If I were you, I'd use your x86 Linux, install docker, install qemu, then use docker buildx to run an aarch64 docker image inside of x86. From inside that image you can install and run PyInstaller and everything it produces will be for ARM. That's a steep learning curve though...
Thankfully I have already climbed that steep curve. I will do this. Thank you @bwoodsend @rokm Your explinations really help me understand what it is that pyinstaller does and how it achieves it.
do I need to run pyinstaller on an arm system for it to work? I thought the purpose of building the bootloader was for cross compatability
Even if you do cross-compile bootloader for target arch, you need to freeze your python script using environment running on that target arch, because PyInstaller needs to collect the python shared library and any other shared libraries / binary extensions from python packages that you use. So you cannot use PyInstaller to "cross-compile" your program for other arches (macOS itself being a bit of an exception there due to
universal2, but even there it really depends).
@rokm I spent a lot of time reading the documentation and that information was never clear IMO. You stated it very clearly, I think it should be in the documentation right, in the section discussing how to build bootloaders.
@rokm I spent a lot of time reading the documentation and that information was never clear IMO. You stated it very clearly, I think it should be in the documentation right, in the section discussing how to build bootloaders.
Yeah, the docs are a bit messy...
But to be fair, there's a note right at the top of "What PyInstaller Does and How It Does It".
And it's also written at the top of the README: "PyInstaller is tested against Windows, Mac OS X, and GNU/Linux. However, it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD, but is not tested against them as part of the continuous integration tests." (I added the emphasis).
I thought the purpose of building the bootloader was for cross compatability
Can you remember what made you think that?
I thought the purpose of building the bootloader was for cross compatability
Can you remember what made you think that?
"The bootloaders architecture defaults to the machine’s one, but can be changed using the --target-arch option – given the appropriate compiler and development files are installed. E.g. to build a 32-bit bootloader on a 64-bit machine, run: "
...
"For more information about cross-building please read on "
...
"Bootloaders can be built for other architectures such as ARM or MIPS using Docker. The bootloader/Dockerfile contains the instructions on how to do this. Open it in some flavour of text previewer to see them"
Hmm, I see where you're coming from. You have made a bit of a logical leap in assuming that PyInstaller's bootloaders can be cross compiled means that PyInstaller itself can cross compile but I suppose that's an understandable missasumption.
For anyone who stumbles across this thread, here is a VERY GOOD TUTORIAL I followed. I did it on my MAC, and virtualized an arm64 environment despite having x86! Now I just have to figure out how to create the needed Dockerfile so that I can use pyinstaller to cross compile my code. Who knows maybe I can even streamline the process! Thank you all in this thread!
heres another great tutorial about how to do it on AWS
ps* much easier to do this with dockerhub, no need to do anything just build!
@bwoodsend you should register for brave BAT rewards so I can tip you :)
Oh, and PyInstaller already ships ARM bootloaders for Linux so you shouldn't need to compile the bootloaders. In Linux arm64 is called aarch64 so if you download a wheel tagged aarch64 linux, that'll run on ARM. You still need an ARM Linux machine (or to virtualise one) though in order to run PyInstaller.
So if I am running on a container that is using arm64 which I have already done, do I need to have python installed in that container to use pyinstaller? or can i just install the wheel and then use pyinstaller on a python file ie main.py?
For anyone who stumbles across this thread, here is a VERY GOOD TUTORIAL I followed. I did it on my MAC, and virtualized an arm64 environment despite having x86!
Oh wow, Docker's macOS and Windows support has really moved on. I had it in my head that it was still purely for virtualising Linux on Linux only and everything else was just a token gesture.
@bwoodsend you should register for brave BAT rewards so I can tip you :)
Actually I've just got a job and the salary is way more than I need so save your tips for someone else. (Personally I think I'd tip the authors of Qemu and Docker buildx.)
So if I am running on a container that is using arm64 which I have already done, do I need to have python installed in that container to use pyinstaller?
Yes, you always need pip to install and Python to run PyInstaller. Use the distribution's package manager to install them, then pip install pyinstaller - pip will automatically install the special ARM variant. Or if you don't mind a slightly bloated build image, use quay.io/pypa/manylinux_2_24_aarch64 as your base image which comes with a range of Python versions preinstalled, gcc and a controlled version of glibc (if you're not already familiar with Linux binaries compatibility - glibc version is crucial).