Error when installing requirement on a M1 MacBook

[fharper]

🐛 Bug

When I try to install the requirements to run doctr on my M1 MacBook, I get an ModuleNotFoundError: No module named '_ctypes'. I installed the additional packages required for macOS users.

https://user-images.githubusercontent.com/1170989/133793566-a8b97c2c-55a7-4129-bd95-a4c6535f9798.mp4

To Reproduce

Steps to reproduce the behavior:

1. pip install -r requirements.txt

Expected behavior

No error when installing the packages.

Environment

DocTR version: N/A TensorFlow version: N/A PyTorch version: N/A (torchvision N/A) OpenCV version: N/A OS: Mac OSX 11.5.2 Python version: 3.7 Is CUDA available (TensorFlow): N/A Is CUDA available (PyTorch): N/A CUDA runtime version: Could not collect GPU models and configuration: Could not collect Nvidia driver version: Could not collect cuDNN version: Could not collect How you installed DocTR (conda, pip, source): I did not, I'm trying to use the code from the repository I cloned

[fg-mindee]

Thanks for reporting it! If M1 MacOS users are getting more common, we indeed have to handle this. However if this is correct, it seems the issue is due to numpy rather than DocTR. In order to confirm this, let's try to get numpy to work and then try again on the same setup :)

@fharper would you mind trying this trick : https://stackoverflow.com/a/65581354 to install numpy? If this works and that doctr installation can be completed afterwards, I will add a warning note for M1-MacOS users in the installation instructions :ok_hand:

[fharper]

By doing

pip install Cython
pip install --no-binary :all: --no-use-pep517 numpy

I am able to install numpy but I cannot install the requirements when running  pip install -r requirements.txt: I still have numpy related issues. See the output:

I'll try to investigate a bit more myself once I have more time.

It doesn't solve the problem, but I completely forgot about Rosetta2 since I didn't have any M1 related issue for a while, but I could probably run doctr like this for now. It's still good to be able to fix the process for M1 users.

If M1 MacOS users are getting more common, we indeed have to handle this.

AFAIK, Apple has about 10% of the laptop market, and they won't build any more devices using an Intel chipset, so it may not be urgent, but certainly something to consider sooner than later.

[fg-mindee]

Thanks for investigating @fharper :pray:

We'll have to think about a way to counter this, the quick fix of adding constraints on the python version is quite heavy on other OS (or even for MacOS considering most users are not necessarily running M1 chips). Most likely, we'll have to ping the authors of the corresponding libraries to investigate this issue!

[fharper]

I'm giving it another try, will keep folks up to date.

Steps

I'll keep my steps here, but keep in mind it's not final, and may change. I installed the dependencies

pip install wheel
pip install Keras-Preprocessing
xcode-select --install

You will also need to have a JDK installed.

~~We need to build bazel as we need version under 4.1 (since we are compiling TensrFlow 2.6.0), and they don't have arm64 build. First download the Bazel source from the https://github.com/bazelbuild/bazel/releases/tag/3.7.2 release dist file. Do not git clone as it doesn't not contain the generated output of the protoc compiler. Be sure that you build bazel from a path with no spaces in folder name!~~

~~env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" --host_jvm_args=--add-opens=java.base/java.nio=ALL-UNNAMED --host_jvm_args=--add-opens=java.base/java.lang=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED bash ./compile.sh~~

~~--host_jvm_args=--add-opens=java.base/java.nio=ALL-UNNAMED & --host_jvm_args=--add-opens=java.base/java.lang=ALL-UNNAMED help reflection issue due to the mplementation of strong encapsulation in v9 as --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED is because internal APIs that are encapsulated in JDK 9 are not accessible at compile time.~~

Let's use MacPorts to install Bazel 3.7.2 on arm64. Once MacPorts installed, run the port command sudo port install bazel-3.7

and now let's install Tensorflow 2.6.0 (Tensorflow needs tensorflow-io which support only up to 2.6.0 for now). Be sure that you build tensorflow from a path with no spaces in folder name!

git clone [email protected]:tensorflow/tensorflow.git
cd tensorflow
git checkout v2.6.0
./configure
/opt/local/bin/bazel-3.7 build //tensorflow/tools/pip_package:build_pip_package
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-2.8.0-cp39-cp39-macosx_11_0_arm64.whl

It's a mix of commands taken from https://docs.bazel.build/versions/4.2.1/install-compile-source.html#bootstrap-bazel, https://www.tensorflow.org/install/source and of my own.

[fg-mindee]

Thanks @fharper :pray:

Have you followed some tutorials or guidelines from a specific resource? If not, is keras preprocessing required to build TF?

[fg-mindee]

I've checked online, and here is what I found about DL framework support on M1 chips:

• TensorFlow: https://towardsdatascience.com/installing-tensorflow-on-the-m1-mac-410bb36b776, https://betterprogramming.pub/installing-tensorflow-on-apple-m1-with-new-metal-plugin-6d3cb9cb00ca
• PyTorch: https://github.com/pytorch/pytorch/issues/47702#issuecomment-965625139

[fharper]

I didn't have time to try to make it work again. I think I've checked those tutorials, but it wasn't working for me. I'll give it another try as soon as I can.

[fg-mindee]

Oh yeah, it's more for tracking purposes :+1: Considering the efforts that people have to put into this to make it work on M1 chips, I wouldn't advise jumping back into this if you're busy!

[fg-mindee]

Hi @fharper :wave:

I think that with #749, you might be able to install everything smoothly! When you have some time, could you try? That would help us close this issue :)

[fharper]

Will try, thanks!

[felixdittrich92]

Hi @fharper can we close this ? :)

[frgfm]

So PyTorch stable now supports Mac M1 architecture! However about TensorFlow, it looks like a plugin is needed: https://developer.apple.com/metal/tensorflow-plugin/

[felixdittrich92]

sounds like a lot of trouble :sweat_smile:

[frgfm]

For now, we should perhaps update the installation instructions in the README & documentation to close this :+1:

[felixdittrich92]

I will try to add some information in the docs. Unfortunately i don't have the option to test it.