pycryptodome icon indicating copy to clipboard operation
pycryptodome copied to clipboard

Failing on M1 Mac - OSError: Cannot load native module 'Crypto.Hash._SHA256'

Open krisun09 opened this issue 3 years ago • 13 comments

I am trying to follow the installation guide for PyCryptodome. But when I try to run Cryptodome / Crypto modules in my code it is giving me the following error.

image

With python3.9, I've tried installing pycryptodome 3.14.1, removing pycryptodome 3.14.1 then install pycryptodomex 3.14.1, removing pycryptodome x3.14.1 then install pycryptodome 3.10.1,

Then I tried to update my python to python3.10 using ARM, and install pycryptodome 3.14.1, which is still giving me the same error. image

Should I try reinstalling python? ;( Any help is appreciated. Thanks in advance.

krisun09 avatar Apr 16 '22 05:04 krisun09

As I do not have an ARM Mac (or any Mac at all), I can only be of little help.

The error message indicates the compiled extensions and your Python installation have different architectures. Do you mind telling which architecture your Python and the installed extension has? You should be able to get it with file /opt/homebrew/lib/python3.10/site-packages/Crypto/Hash/_SHA256.abi3.so and file $(which python3).

Varbin avatar Apr 21 '22 16:04 Varbin

As I do not have an ARM Mac (or any Mac at all), I can only be of little help.

The error message indicates the compiled extensions and your Python installation have different architectures. Do you mind telling which architecture your Python and the installed extension has? You should be able to get it with file /opt/homebrew/lib/python3.10/site-packages/Crypto/Hash/_SHA256.abi3.so and file $(which python3).

I am having a near identical issue (but failing to import _raw_ecb native module). PyCryptoDome was installed as part of other project dependencies (using Poetry over pip), and have the same errors coming back as the OP.

Outputs of your suggested commands: file /Users/XXXX/environments/grid-actions/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_ecb.abi3.so

/Users/XXXX/environments/grid-actions/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_ecb.abi3.so: Mach-O 64-bit bundle x86_64

file $(which python3) (Run inside my virtualenv)
/Users/XXXX/environments/grid-actions/bin/python3: Mach-O 64-bit executable arm64

Will-Gaines avatar Aug 09 '22 15:08 Will-Gaines

Some users of my application are complaining about the same problem.

xilopaint avatar Aug 10 '22 02:08 xilopaint

I also have this problem with Python 3.8.12, Poetry, on Apple M1

j1nma avatar Aug 17 '22 16:08 j1nma

same problem here, python 3.10

Luke7389 avatar Aug 18 '22 21:08 Luke7389

resolved by downgrading to python 3.9, in my case I was using a function that uses a module called pyteal, downgrading forced me to use a different version of petal and now everything works as expected

Luke7389 avatar Aug 18 '22 21:08 Luke7389

resolved by downgrading to python 3.9

This is the traceback from one of them:

Traceback (most recent call last):
  File "/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/alfred_pdf_tools.py", line 48, in <module>
    from PyPDF2 import PdfReader, PdfWriter, PdfMerger, PageRange, PageObject, errors
  File "/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/PyPDF2/__init__.py", line 10, in <module>
    from ._encryption import PasswordType
  File "/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/PyPDF2/_encryption.py", line 60, in <module>
    from Crypto.Cipher import AES, ARC4  # type: ignore[import]
  File "/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/Crypto/Cipher/__init__.py", line 27, in <module>
    from Crypto.Cipher._mode_ecb import _create_ecb_cipher
  File "/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/Crypto/Cipher/_mode_ecb.py", line 35, in <module>
    raw_ecb_lib = load_pycryptodome_raw_lib("Crypto.Cipher._raw_ecb", """
  File "/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/Crypto/Util/_raw_api.py", line 309, in load_pycryptodome_raw_lib
    raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Not found '_raw_ecb.cpython-39-darwin.so', Cannot load '_raw_ecb.abi3.so': dlopen(/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/Crypto/Util/../Cipher/_raw_ecb.abi3.so, 0x0006): tried: '/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/Crypto/Util/../Cipher/_raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.F5626124-7D80-45A1-A349-E413ADB69042/Crypto/Cipher/_raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), Not found '_raw_ecb.so'

xilopaint avatar Aug 18 '22 21:08 xilopaint

@xilopaint although they might be related, you are now receiving a different error, which looks like the one Will is receiving, in my case it worked simply downgrading. I used a clean python installation

Luke7389 avatar Aug 18 '22 22:08 Luke7389

Hi, I'm currently running into this issue myself. Is there a clear idea on why the x86 shared object is being installed instead of the arm64 one? Is there a way to force the install to use the arm64 version? It seems that this isn't version specific to Pycryptodome, but python. Things are working somehow when I am on python 3.9 vs python 3.10. @Legrandin your help would be much appreciated here!

pyrito avatar Jan 05 '23 21:01 pyrito

Hi, I have the same error. For you investigation, I managed to use 3.15.0 version with conda from a fresh new env, after installing web3 lib : pip install web3 pip uninstall pycryptodome(v3.16.0) conda install pycryptodome (v3.15.0) It was not the last version... why?

jeugregg avatar Jan 09 '23 07:01 jeugregg

I am also having the same problem. I get an error in lambda. I installed the following. docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.9" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.9/site-packages/; exit" I also tried using docker with amazonlinux:latest but it does not work.

Cannot load native module 'Crypto.Cipher._raw_ecb': Not found '_raw_ecb.cpython-39-x86_64-linux-gnu.so', Cannot load '_raw_ecb.abi3.so': /opt/python/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_ecb.abi3.so: cannot open shared object file: No such file or directory, Not found '_raw_ecb.so'

Does anyone know anything about this?

Rikuto7 avatar Feb 23 '23 12:02 Rikuto7

I have same issue here. I am trying to make layer for aws sam project. I am using macbook m1. "Cannot load native module 'Crypto.Hash._MD5': Not found '_MD5.cpython-39-aarch64-linux-gnu.so', Cannot load '_MD5.abi3.so': /opt/python/Crypto/Util/../Hash/_MD5.abi3.so: invalid ELF header, Not found '_MD5.so'"

EnkhAmar avatar Apr 24 '23 15:04 EnkhAmar

I also have the same problem as @Rikuto7 . I have an application which depends on pycryptodome and I try to run it in a docker container. I use ubuntu focal as the base of my docker image. I install pycryptodome with apt. Once I start my application I get the following error:

from Cryptodome.Hash import MD4
  File "/usr/lib/python3/dist-packages/Cryptodome/Hash/MD4.py", line 54, in <module>
    _raw_md4_lib = load_pycryptodome_raw_lib(
  File "/usr/lib/python3/dist-packages/Cryptodome/Util/_raw_api.py", line 259, in load_pycryptodome_raw_lib
    raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
OSError: Cannot load native module 'Cryptodome.Hash._MD4': 

What I figured out is that when pycryptodome tries to collect the C file extensions suffixes the .cpython-38-x86_64-linux-gnu.so suffix is not showing up.

This is the 3.6.1 pycryptodome part which handles the lib loading:

if sys.version_info[0] <= 3 or \
   (sys.version_info[0] == 3 and sys.version_info[1] <= 3):

    import imp
    extension_suffixes = []
    for ext, mod, typ in imp.get_suffixes():
        if typ == imp.C_EXTENSION:
            extension_suffixes.append(ext)

else:
    from importlib import machinery
    extension_suffixes = machinery.EXTENSION_SUFFIXES

It doesn't matter which branch get executed the .cpython-38-x86_64-linux-gnu.so extension suffix is missing, although when I start the interpreter and run imp.get_suffixes() or machinery.EXTENSION_SUFFIXES the suffixes is there.

When I modify the lib and include the suffix in the initial extension_suffixes or append to it the .so get loaded fine

I don't know if the two cases (docker, Mac M1) is related but very similar. I also know that my lib version a bit out dated but I have strict version rules and cannot update but the lib loading logic hasn't changed that much as I can see.

python version is 3.8.10 pycryptodome is 3.6.1

gorosz avatar Jun 16 '23 07:06 gorosz