kivy
ctypes.pythonapi issues; getting AttributeError: undefined symbol
Using the new python3 toolchain, I am trying to use PyCryptodome (3.7+) as a dependency.
As in https://github.com/kivy/python-for-android/pull/1685, PyCryptodome >= 3.6.4 runs into compilation issues; I am trying to fix these too, though this is not that relevant here.
PyCryptodome >=3.6.0 crashes at runtime (since commit https://github.com/Legrandin/pycryptodome/commit/f5aa2c1618e97b6d773172fdd07794a0a6f05905). So actually the currently pinned version in the recipe does not work: https://github.com/kivy/python-for-android/blob/80e4f059c1ee0da48a7c85167087dfe5928ac395/pythonforandroid/recipes/pycryptodome/init.py#L5
The issue at runtime is with ctypes.
Say I have a main script that just does the following (https://github.com/Legrandin/pycryptodome/blob/95ccce7ae82d3a36f1a8652dd2c645222d0128dd/lib/Crypto/Util/_raw_api.py#L200):
import ctypes
ctypes.pythonapi.PyObject_GetBuffer
This works with cpython on my laptop, but with the p4a-compiled python on Android it fails:
06-14 19:06:27.053 15246 15274 I python : Android kivy bootstrap done. __name__ is __main__
06-14 19:06:27.053 15246 15274 I python : AND: Ran string
06-14 19:06:27.053 15246 15274 I python : Run user program, change dir and execute entrypoint
06-14 19:06:27.092 15246 15274 I python : Traceback (most recent call last):
06-14 19:06:27.092 15246 15274 I python : File "/home/user/wspace/electrum/.buildozer/android/app/main.py", line 84, in <module>
06-14 19:06:27.093 15246 15274 I python : File "/home/user/wspace/electrum/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 369, in __getattr__
06-14 19:06:27.093 15246 15274 I python : File "/home/user/wspace/electrum/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 374, in __getitem__
06-14 19:06:27.094 15246 15274 I python : AttributeError: undefined symbol: PyObject_GetBuffer
I have also tried to access some other attributes of ctypes.pythonapi, such as Py_IncRef, which raises the same exception.
Any idea what's going on here? Why is ctypes.pythonapi non-functional?
Not sure what's going on here, but it isn't impossible that this symbol is missing for some environment-related reason. I'll try to look into it.
Presumably this has worked for some people though, @AndreMiras any thoughts?
I wanted to chime in and say I ran into the same problem, with a nearly identical stack trace. Any help is appreciated!
I'm currently using pycryptodome==3.4.6 for two of my projects without issue.
I'll give 3.6.3 a try then
I've tried playing with some pycryptodome primitives with the following code.
from typing import List
import Crypto
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes
def encrypt(key: bytes, data: bytes) -> List[bytes]:
"""
Given a key and data returns a list containing nonce, tag and ciphertext.
"""
cipher = AES.new(key, AES.MODE_EAX)
ciphertext, tag = cipher.encrypt_and_digest(data)
encrypted = [x for x in (cipher.nonce, tag, ciphertext)]
return encrypted
def decrypt(key: bytes, data: bytes) -> bytes:
"""
Given a key and data (nonce, tag and ciphertext), returns decrypted data.
"""
nonce, tag, ciphertext = data
cipher = AES.new(key, AES.MODE_EAX, nonce)
decrypted = cipher.decrypt_and_verify(ciphertext, tag)
return decrypted
def main():
print(f'Crypto.version_info: {Crypto.version_info}')
key = get_random_bytes(16)
data = b'Testing pycryptodome on Python for Android'
print(f'data: {data}')
encrypted = encrypt(key, data)
print(f'encrypted: {encrypted}')
decrypted = decrypt(key, encrypted)
print(f'decrypted: {decrypted}')
assert data == decrypted
if __name__ == '__main__':
main()
And it produced the following output on device.
08-24 00:28:10.660 14861 15109 I python : Android kivy bootstrap done. __name__ is __main__
08-24 00:28:10.660 14861 15109 I python : AND: Ran string
08-24 00:28:10.660 14861 15109 I python : Run user program, change dir and execute entrypoint
08-24 00:28:13.172 14861 15109 I python : Crypto.version_info: (3, 6, 3)
08-24 00:28:13.173 14861 15109 I python : data: b'Testing pycryptodome on Python for Android'
08-24 00:28:13.190 14861 15109 I python : encrypted: [b'\x7fJ\xa9P\x00-\x96G\x8d\x1d4=M3B\xc0', b'\xa5K\xc5@\xb1\xed\x88@\xa2\xf15\xb0\x81\xe4Sp', b'2S\xbfo\x85\x04\x8c\x15\x92h\x83\xaeQ\xeem\x93\xe9 |\x11Z|\x82=\n~\x99\xbd\xe6\xca\x0b\xbd<|\x81`\x94!I$\xdaX']
08-24 00:28:13.195 14861 15109 I python : decrypted: b'Testing pycryptodome on Python for Android'
08-24 00:28:13.195 14861 15109 I python : Python for android ended.
So I guess next up is to mess up with ctypes directly
And I could confirm the issue with ctypes.
08-24 00:52:08.844 16982 17327 I python : Run user program, change dir and execute entrypoint
08-24 00:52:08.866 16982 17327 I python : ctypes.__version__: 1.1.0
08-24 00:52:08.909 16982 17327 I python : Traceback (most recent call last):
08-24 00:52:08.909 16982 17327 I python : File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/app/main.py", line 10, in <module>
08-24 00:52:08.911 16982 17327 I python : File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/app/main.py", line 6, in main
08-24 00:52:08.912 16982 17327 I python : File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 369, in __getattr__
08-24 00:52:08.913 16982 17327 I python : File "/home/andre/workspace/python-for-android/issues/1866/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ctypes/__init__.py", line 374, in __getitem__
08-24 00:52:08.914 16982 17327 I python : AttributeError: undefined symbol: PyObject_GetBuffer
08-24 00:52:08.914 16982 17327 I python : Python for android ended.
Code was:
import ctypes
def main():
print(f'ctypes.__version__: {ctypes.__version__}')
print(f'ctypes.pythonapi.PyObject_GetBuffer: {ctypes.pythonapi.PyObject_GetBuffer}')
if __name__ == '__main__':
main()
Edit: This is in the C-API https://docs.python.org/3/c-api/buffer.html#c.PyObject_GetBuffer https://github.com/python/cpython/blob/3.7/Objects/abstract.c#L353 hence should be part of libpython3.7.so. And confirmed on Ubuntu 18.04:
readelf -s /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7.so | grep PyObject_GetBuffer
1857: 000000000027b960 68 FUNC GLOBAL DEFAULT 12 PyObject_GetBuffer
And in the p4a build.
readelf -s .buildozer/android/platform/build/dists/myapp/build/intermediates/jniLibs/debug/armeabi-v7a/libpython3.7m.so | grep PyObject_GetBuffer
315: 0003b8e4 84 FUNC GLOBAL DEFAULT 11 PyObject_GetBuffer
So it's as if we were not linking to it.
Initially, in pycryptodome, ctypes is used as fallback to cffi, because of the "ImportError: CFFI with optimize=2 fails due to pycparser bug."
Is there any way to make cffi work in python-for-android, so that pycryptodome can work despite ctypes issues? I've seen in the current master some cmd line params like "no_compile_pyo" and "optimize_python", but I don't really get how/when they are used in the compilation toolchain (the pycryptodome files I find in the build directory are "pyc" and not "pyo", but they seem to be stripped from their docstrings anyway).
On the other hand, the fallback from cffi to ctypes, above, looks at sys.flags.optimize only, not the real state of some files... is there any way to make p4a run the python interpreter in non optimized mode, to see if it solves problems?
EDIT: I forced both flags in p4a to skip bytecode optimization, but files compiled into "_python_bundle/" are still without docstring, and anyway the python interpreter seems to be forced to use PYTHONOPTIMIZED=2 in multiple parts of the toolchain... so cffi seems unusable.
This is not related to the main ctypes.pythonapi issue (which has maybe existed for very long), but if anyone needs to compile the latest pycryptodome for android, commenting out a few "Setenv("PYTHONOPTIMIZE", "2")" in start.c and in PythonActivity.java files allows it to use cffi and not hit that ctypes bug.
For what it's worth, I've just noticed that other cross-compilation attempts to android did precisely patch "pythonapi" in ctypes, maybe it's what's missing in P4A?
The post: http://mdqinc.com/blog/2011/09/cross-compiling-python-for-android/
The patch: https://bitbucket.org/gabomdq/ignifuga/raw/tip/tools/patches/python.android.diff
I'm facing the exact same issue. In which directory can I find the start.c that you've modified? Also what do I have to do exactly to fix the bug?
Also @AndreMiras does downgrading pycryptodome help or does downgrading ctypes help (if that is possible)
P.S. I am using pycryptodomex, but I don't think that should make a difference since the only difference I am aware of is the namespace.
@ShreeSinghi my workaround works by searching the line Setenv("PYTHONOPTIMIZE", "2") in the source tree of P4A (there are multiple occurrences) and removing these lines ; this allows non-optimized mode for python compiled files, and thus makes ctypes use a different backend, without hitting the bug.
Some other project's patch of python to support ctypes on Android is now here : https://github.com/gabomdq/ignifuga/blob/master/tools/patches/python.android.diff
Copy here in case it disappears again python.android.diff.zip
For what it's worth : I've monkey-patched ctypes from the beginning of my main app, to force the loading of the proper python DLL, and it seems to work fine (PyCryptodome 3.9.9 works with this); thus no need to rely on CFFI by disabling PYTHONOPTIMIZE in P4A.
import ctypes, sys
ctypes.pythonapi = ctypes.PyDLL("libpython%d.%d.so" % sys.version_info[:2]) # replaces ctypes.PyDLL(None)
What would be the proper way to integrate this into P4A though?
With this I could update the PyCryptodome recipe.