Briefcase can't install Python packages with binary modules on mobile platforms.

[freakboy3742]

Describe the bug

Briefcase is able to package any an app that has a dependency that is a pure-Python package (i.e., a package that contains only Python code).

However, if you're targeting a mobile platform (iOS or Android), and the Python module has a binary component (i.e., it contains a C module), any attempt to import that module will fail, causing the app to crash.

Desktop platforms (macOS, Windows and Linux) are not affected by this problem.

To Reproduce Steps to reproduce the behavior:

1. Set up a hello world application
2. Add a package with a binary dependency (e.g., numpy or pillow) to the requires list, and add an import for that code (e.g., import numpy or import pil)
3. Deploy the app to iOS or Android
4. Try to run the app.

Expected behavior

The import should succeed and the app should continue to run.

Environment:

• Operating System: iOS or Android
• Python version: Any
• Software versions:
  • Briefcase: Any

Additional context

This isn't purely a Briefcase issue; pip and PyPI doesn't currently have any support for mobile platforms.

[somebodyLi]

is there any other possible idea to fix ? Is there an alternative to this package(pycryptodome)？

[freakboy3742]

I don't know enough about pycryptodome to comment; however, OpenSSL is available as a precompiled library, so any python binding to openSSL crypto primitives should work.

[somebodyLi]

pycryptodomex is a very important crypto library in python ? Is it possible to add it to template like OpenSSL as a precompiled library ??? it's troubles me for many times

[freakboy3742]

It is possible to add any library you want to the support package by forking and modifying the build process; however, we're unlikely to include it in the official package. OpenSSL is included because it's needed to compile the Python standard library.

The fix here isn't to add more and more binary libraries to the support package; it's to fix the underlying problem of binary wheels for mobile platforms.

[sergiysavelyev]

Sorry, I am a little confused and hope you can clarify my doubt. The subject says - "Briefcase can't install Python packages with binary modules". The example given in Aug 13th post tells that expected behavior when including "import numpy" is "The import should succeed and the app should continue to run." So, does briefcase include numpy dependency in Android apk? I built an app that uses numpy; when I debug the apk in Android Studio, the log shows - "ModuleNotFoundError: No module named 'numpy'", which leads me to believe that Briefcase does not handle numpy dependency. Please advise.

[freakboy3742]

@sergiysavelyev I think you may be misreading the ticket. "Expected behavior" is what should happen, but due to a bug, does not happen currently.

You can specify any PyPI package as a dependency (pure python or binary); and briefcase will include that package in the bundled app. However, on mobile platforms, apps that use binary dependencies will crash when the binary module is imported. This will surface as a "no module named 'numpy'" error because the import will not succeed, as the binary module will fail to load - the module lookup process will be looking for an architecture-specific binary file that doesn't exist.

The same app will work fine on desktop platforms. The issue only affects iOS and Android.

[sergiysavelyev]

Got it, thanks. Still a pity not to be able to use one of the most popular python libraries in Beeware projects on Android devices.

[freakboy3742]

Oh, I completely agree. It's something we want to fix, and we broadly know how to fix it - but it's not a trivial fix. We need to find the time and resources to do the work, or we need a volunteer to step up and do the work.

[renefritze]

Is this limitation mentioned in the docs? I've run through the tutorials and only saw the "platform notice". An explicit note would have saved me a good chunk of time :disappointed:

[acheronfail]

I would like to be able to use the dependency netifaces (on Android), but due to this bug I can't.

The netifaces dependency does support ARM systems though, so I guess this may be a matter of somehow getting Briefcase to build dependencies for the correct architecture? :thinking:

I tried to do a dirty hack of installing netifaces in Termux, copying its files out of the site-packages folder there and replacing them in the app I created with Briefcase, but that ended up with a dlopen error... :sweat_smile:

[Tadashi-Hikari]

This could be my inexperience with Python, but can we access the source for those binaries? If briefcase integrates with Gradle at all, you can run the C/C++ source code through the Android NDK and use Gradle to build the binaries for the target devices. This is roughly how the Android NDK build system works now

[freakboy3742]

@Tadashi-Hikari Yes, the source is available; and the build instructions can be inferred as well (they're usually part of the setup.py configuration - although those definitions can sometimes be complex).

This is something that could be integrated into the Gradle config on a per-project basis; however, it's probably going to be easier to do an external build of the dependency to compile an actual library - much the same way that binary wheels are currently compiled for desktop platforms - and then deploy the android/iOS binary wheel into the app.

This would also allow projects with complex build processes (like numpy) to manage the compilation process and publish artefacts, rather than requiring every user to have a full development toolchain for that library.

[Tadashi-Hikari]

I'm not super familiar with the build process in general, outside of general familiarity with make and autoconf. What would I need to look in to and learn in order to get started on this process?

[freakboy3742]

I'd suggest picking a simple binary module (https://pypi.org/project/pyspamsum/ is one that I maintain), and follow the trail from there. Try to compile that module manually in a way that is compatible with the mobile platform; and work out how to install the module so that Android can use it at runtime; then work out how to modify setuptools and pip to reproduce those manual processes.

As a heads up - this is going to be a complex project (There's a good reason we haven't done it to date). Unfortunately, it's also a project where there aren't a lot of people who know the systems well enough to be able to help. But, if you're up for a challenge, this would be immensely helpful.

The other heads up: this will be a lot easier on Android than iOS (at least for now); iOS has another technical problem in that we need to work out how to get dynamic module loading working on iOS. To date, BeeWare's iOS support is 100% statically loaded. This is for historical reasons; prior to iOS 8, that was the only option. Dynamic loading is apparently now possible, but we need to make our build compatible with those changes.

[Tadashi-Hikari]

So long as Android stays the dominant mobile platform, I'm 100% down to look in to this over time. I've been working on an on device mobile assistant (degoogled) and I've finally gotten STT, TTS, and NLP working without network connection. Having Python & C/C++ modules would make things a lot easier for me as I could have more ML libraries and tools onboard. Complexity doesn't scare me, it just takes a lot of time. I can't say I really have any interest in doing this for iOS though.

Does it matter which binary module I go with? I'm inclined to start with something like numpy or pandas, honestly. am I looking more in to the existing build system itself?

[freakboy3742]

Totally understood regarding platform choice - I'm much the same but with platforms reversed. Having solid contributions for any platform is a win, so I won't be picky about someone who doesn't want to work on improving a platform they don't use.

As for which binary module - essentially no; although I'd advise against starting with numpy. Those are large and complex beasts, and I suspect you'll get a lot more traction starting with something smaller and working up to numpy.

I'd suggest there's three progress targets to aim for:

• Simple C module, no external dependencies (like pyspamsum; a single C file that needs to be compiled)
• Moderate C module with an external dependency (cryptography, maybe? Something that is a small number of C files, with an external .SO file that is part of the compile dependencies
• Complex C modules (numpy, pandas, scikit-learn et al)

[Tadashi-Hikari]

When it comes to simple C modules, the Android build system is pretty straightforward. It's not even too bad for moderate C modules if you know what's what with make files (ndk-make and Cmake). Your recommendation is strictly to work within briefcase and not work to merge Gradle and briefcase in some way? Is there a briefcase plugin for Android Studio?

I spend a lot of time fighting Android as a system as is since I prefer the Unix way of development, I'd like to avoid fighting the grain of Android Studio if possible. However, if that's out of the scope of briefcase then I'll learn the briefcase way a bit more

[freakboy3742]

No, there isn't an Android Studio plugin for BeeWare. We use the command-line tools to build a stub grade project and start the simulator, but we don't use the Studio GUI at all.

This isn't a briefcase problem per se; it's an issue with getting setuptools and pip to do something compatible with Android. Briefcase expects pip install module to Just Work. The challenge is to get setuptools to compile Android artefacts at all; and to get pip to install those artefacts in a way that they can be used at runtime. There might be some sort of change needed in briefcase to make this work; but in an ideal world, briefcase is only an automation tool, not something that is harbouring any particularly smart logic.

[Tadashi-Hikari]

Ah, I see. That changes the scope and focus of the issue. I'll dig around to see what I can find out

[mansourmoufid]

I am investigating using Toga to make an Android app. I stumbled on this issue. I might have a solution: github.com/eliteraspberries/python-androidenv But it needs a small patch to Python's distutils.

I've got NumPy building, I'll check back in when I've actually got it running on my phone.

[mansourmoufid]

So here's what I have so far:

diff --git a/setup.cfg b/setup.cfg
index cdea195..67a41a5 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -62,6 +62,7 @@ install_requires =
     GitPython >= 3.0.8
     dmgbuild >= 1.3.3; sys_platform == "darwin"
     Jinja2
+    androidenv >= 0.3.1
 
 [options.packages.find]
 where = src
diff --git a/src/briefcase/commands/create.py b/src/briefcase/commands/create.py
index 41f33cf..c2cd7cb 100644
--- a/src/briefcase/commands/create.py
+++ b/src/briefcase/commands/create.py
@@ -387,14 +387,17 @@ class CreateCommand(BaseCommand):
         """
         if app.requires:
             try:
+                pip = [sys.executable, "-m", "pip"]
+                options = [
+                    "--upgrade",
+                    "--no-user",
+                    "--target={}".format(self.app_packages_path(app)),
+                ]
+                if self.platform == "android":
+                    pip = [sys.executable, "-m", "androidenv"] + pip
+                    options += ["--no-binary", ":all:"]
                 self.subprocess.run(
-                    [
-                        sys.executable, "-m",
-                        "pip", "install",
-                        "--upgrade",
-                        "--no-user",
-                        "--target={}".format(self.app_packages_path(app)),
-                    ] + app.requires,
+                    pip + ["install"] + options + app.requires,
                     check=True,
                 )
             except subprocess.CalledProcessError:

Basically, prepend "python -m androidenv" to the pip command.

The option "--no-binary :all:" is necessary otherwise pip tries to use wheels for the host (macOS):

briefcase create android
...
[helloworld] Installing dependencies...
Collecting numpy
  Using cached numpy-1.21.4-cp37-cp37-macosx_11_0_arm64.whl

With that, the app builds fine, then gives this error at run time:

briefcase run android
...
12-06 11:53:15.078 30435 30435 D MainActivity: Python.run() start
12-06 11:53:15.079 30435 30435 D Python  : Running 'helloworld' as __main__...
12-06 11:53:17.050 30435 30435 E Python  : Application quit abnormally!
...
12-06 11:53:17.054 30435 30435 E Python  : ImportError: dlopen failed: cannot locate symbol "PyBool_Type" referenced by "/data/data/com.example.helloworld/files/python/user_code/app_packages/numpy/core/_multiarray_umath.so"...

The symbol PyBool_Type should be in libpython.so, and it is:

(bee-venv) % python -m androidenv /bin/sh
(bee-venv) % llvm-nm -D "android/gradle/Hello World/app/libs/arm64-v8a/libpython3.7m.so" | grep PyBool
000000000007b5f4 T PyBool_FromLong
0000000000386b38 D PyBool_Type

I don't know if _multiarray_umath.so should be linked to libpython.so? It's not:

(bee-venv) % python -m androidenv /bin/sh
(bee-venv) % llvm-objdump -p "android/gradle/Hello World/app/src/main/assets/python/app_packages/numpy/core/_multiarray_umath.so" | grep NEEDED
  NEEDED       libm.so
  NEEDED       libdl.so
  NEEDED       libc.so

But the macOS version isn't either and it imports fine:

(bee-venv) % python
Python 3.7.12 (default, Dec  5 2021, 23:37:56) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> exit()
(bee-venv) % otool -L bee-venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.so
bee-venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.so:
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

I don't understand dlopen on Android well enough. I'll read more documentation and try again later.

Edit: Hello @DanAlbert, @dimitry-, @enh. Could you please help?

Update: Linking to libpython.so did the trick.

[pmp-p]

hi @eliteraspberries you could bruteforce add libpython3.7m to the needed list in the elf header manually with patchelf, it's not pretty but it works fine like here https://github.com/pmp-p/pydk/blob/8a058a8cadc2f91c85492a95555e30eaf3eaedff/cross-modules.sh#L188

nb use that one https://github.com/NixOS/patchelf/releases instead of an older system one

[mansourmoufid]

hi @eliteraspberries you could bruteforce add libpython3.7m to the needed list in the elf header manually with patchelf, it's not pretty but it works fine like here https://github.com/pmp-p/pydk/blob/8a058a8cadc2f91c85492a95555e30eaf3eaedff/cross-modules.sh#L188

That did work. But I went back and figured out how to link to the libpython.so included in briefcase. Now everything builds and runs fine.

[mansourmoufid]

I modified the hello world tutorial like so:

        import numpy
        label = toga.Label(
            'Hello NumPy, {}'.format(numpy.random.random()),
            style=Pack(padding=(10, 10))
        )
        main_box.add(label)

and the result:

Pull request incoming...

[dillonhaughton]

How is this problem coming along in iOS? any hopes of non python containing modules?

[freakboy3742]

@dillonhaughton It's on the list :-) I can't make any promises on when it's likely to be addressed, but it's definitely high on the priority list.

[Py-CodeTech]

How did you do it ??? I just keep getting the error

[freakboy3742]

Good news - we've been able to make significant progress on this.

Android

Android support for binary modules will be addressed by switching to Chaquopy as a base for our Android work. This will only require a change in Gradle template; a draft of this template change is available: beeware/briefcase-android-gradle-template#52

At present, Chaquopy only supports Python 3.8; @mhsmith is working on adding support for at least Python 3.9 and 3.10; 3.11 support should be available very shortly after it is officially released. Given Python 3.7 is 12 months away from EOL, and there are significant changes in the CPython API between 3.7 and 3.8, we will deprecate support for Python 3.7 a little early.

iOS

On iOS, we need to make a number of changes to the support packages, templates, and Briefcase itself. These changes are:

• beeware/Python-Apple-support#161
• beeware/Python-Apple-support#162
• beeware/Python-Apple-support#163
• beeware/Python-Apple-support#164
• beeware/Python-support-testbed#2
• beeware/briefcase-iOS-Xcode-template#7
• beeware/briefcase-macOS-Xcode-template#10
• beeware/briefcase-macOS-app-template#8
• #842
• #849
• #862
• beeware/skep#4

The Apple support packages have been ported to Python 3.11 (RC1), 3.10, 3.9, and 3.8. We will drop support for Python 3.7 due to changes in the CPython API.

Third party binary modules

Anaconda.org provides a facility for users to set up custom PyPI-compatible repositories of packages; we're planning to use a BeeWare repo to serve iOS packages.

Chaquopy currently maintains an independent PyPI-like package repository that contains pre-built packages for the most popular Android binary packages. It's possible this might move to the Anaconda repository; however, functionally, it doesn't make that much difference.

As the build process for binary modules is... complicated..., we're going to write up some documentation so that third parties can build their own third party binaries.

[mhsmith]

FYI, here's a list of Chaquopy's current binary Python packages, sorted by number of downloads in the last 6 months:

zcat chaquo-access.log.{2..182}.gz | grep -E 'GET /pypi-7.0/.*\.whl HTTP.*pip/' | cut -d' ' -f8 | grep -v 'chaquopy-' | sed -E 's|/pypi-7.0/(.*)/.*|\1|' | sort | uniq -c | tr -s ' ' | sort -nr

 7568 numpy
 3516 matplotlib
 3486 scipy
 3376 kiwisolver
 2948 pillow
 2415 opencv-python
 1858 pandas
 1541 scikit-learn
 1017 regex
 897 grpcio
 873 pywavelets
 844 tensorflow
 807 h5py
 732 scikit-image
 718 murmurhash
 688 dlib
 664 torch
 613 cffi
 513 opencv-contrib-python
 464 opencv-contrib-python-headless
 413 numba
 409 llvmlite
 334 torchvision
 296 cryptography
 281 brotli
 278 pycryptodomex
 254 lxml
 185 shapely
 180 multidict
 172 yarl
 163 aiohttp
 156 frozenlist
 148 tflite-runtime
 144 spacy
 144 blis
 143 srsly
 143 preshed
 143 cymem
 142 thinc
 121 cytoolz
 121 backports-zoneinfo
 112 pycryptodome
 109 tokenizers
 96 soundfile
 89 psutil
 84 google-crc32c
 81 pynacl
 78 opencv-python-headless
 76 coincurve
 62 ujson
 62 sentencepiece
 58 bcrypt
 50 ruamel-yaml-clib
 48 xgboost
 44 greenlet
 41 statsmodels
 39 pyzmq
 39 gevent
 39 bitarray
 38 editdistance
 32 lru-dict
 32 gensim
 28 netifaces
 25 ta-lib
 21 ephem
 20 spectrum
 19 zstandard
 19 cvxopt
 18 rawpy
 16 pyzbar
 16 pysha3
 16 pycurl
 16 argon2-cffi
 15 pycrypto
 14 typed-ast
 12 pycares
 9 wordcloud
 8 twisted
 8 scandir

[freakboy3742]

All the PRs needed to support binary modules on iOS and Android have now been merged, so I'm going to close this ticket. The changes will be in the next release of Briefcase (v0.3.10), due for release in the next day or two (pending final release testing)