federated
federated copied to clipboard
tensorflow
Which version is ready for Mac with M1 Apple Silicon Chip?
Hello Everyone:
I tried to intall tff on my Mac but met a problem.
Preparing wheel metadata ... done
Collecting h5py
Using cached h5py-3.2.1.tar.gz (368 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... |^[[-^[[done
Preparing wheel metadata ... done
ERROR: Cannot install tensorflow-federated==0.1.0, tensorflow-federated==0.10.0, tensorflow-federated==0.10.1, tensorflow-federated==0.11.0, tensorflow-federated==0.12.0, tensorflow-federated==0.13.0, tensorflow-federated==0.13.1, tensorflow-federated==0.14.0, tensorflow-federated==0.15.0, tensorflow-federated==0.16.0, tensorflow-federated==0.16.1, tensorflow-federated==0.17.0, tensorflow-federated==0.18.0, tensorflow-federated==0.2.0, tensorflow-federated==0.3.0, tensorflow-federated==0.4.0, tensorflow-federated==0.5.0, tensorflow-federated==0.6.0, tensorflow-federated==0.7.0 and tensorflow-federated==0.9.0 because these package versions have conflicting dependencies.
The conflict is caused by:
tensorflow-federated 0.18.0 depends on tensorflow~=2.4.0
tensorflow-federated 0.17.0 depends on tensorflow~=2.3.0
tensorflow-federated 0.16.1 depends on tensorflow-addons~=0.10.0
tensorflow-federated 0.16.0 depends on tensorflow-addons~=0.10.0
tensorflow-federated 0.15.0 depends on tensorflow-addons~=0.10.0
tensorflow-federated 0.14.0 depends on tensorflow-addons~=0.9.1
tensorflow-federated 0.13.1 depends on tensorflow~=2.1.0
tensorflow-federated 0.13.0 depends on tensorflow~=2.1.0
tensorflow-federated 0.12.0 depends on tensorflow~=2.1.0
tensorflow-federated 0.11.0 depends on tensorflow-addons~=0.6.0
tensorflow-federated 0.10.1 depends on tensorflow-addons~=0.6.0
tensorflow-federated 0.10.0 depends on tensorflow-addons~=0.6.0
tensorflow-federated 0.9.0 depends on tfa-nightly
tensorflow-federated 0.7.0 depends on tf-nightly
tensorflow-federated 0.6.0 depends on tf-nightly
tensorflow-federated 0.5.0 depends on tf-nightly
tensorflow-federated 0.4.0 depends on tensorflow~=1.13
tensorflow-federated 0.3.0 depends on tensorflow~=1.13
tensorflow-federated 0.2.0 depends on tensorflow~=1.13
tensorflow-federated 0.1.0 depends on tensorflow>=1.13.0rc2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
I tried to install tff with Python 3.8.7 or Python 3.9.1, installed with pyenv, the dependency conflict error raised.
Expect anyone's help. Thanks a lot!
BTW:
I also post the issue on StackOverflow for more help.
I make a comparation between my two Macs with x86 and ARM.
Both of them create a virtual environment with Python 3.8.7 and run:
pip install --upgrade tensorflow-federated-nightly
The installation succeed on the x86 computer. But raise error on the ARM one.
...
tensorflow-federated-nightly 0.16.1.dev20200826 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200822 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200821 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200820 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200819 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200818 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200816 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200815 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200814 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200813 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200811 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200808 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200807 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200806 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200805 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200804 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200803 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200730 depends on tfa-nightly
tensorflow-federated-nightly 0.16.1.dev20200729 depends on tfa-nightly
tensorflow-federated-nightly 0.15.0.dev20200728 depends on tfa-nightly
tensorflow-federated-nightly 0.15.0.dev20200727 depends on tfa-nightly
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
Hi, I met the same problem. Have you solved that?
Hello.
Sadly, I have no solution.
Currently we have tfa-nighlty support for M1 e.g. as you can see:
https://pypi.org/project/tfa-nightly/0.15.0.dev20211014164307/#files
For TF-Addons 0.14 -> TF 2.6 instead is on hold: https://github.com/tensorflow/addons/pull/2583
Hi, I met the same problem. Have you solved that?
Hello.
Sadly, I have no solution.
Hi, have you solved that?
This is still an issue today. I gave it a try to install everything using Rosetta and an x86_64 version of Python, but in the end I got a hardware related error when importing tensorflow in python.
I tried compiling tensorflow_federated on mac m1. I was able to compile the pip package, but when I try importing python crashes
#python3
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:00:33)
[Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow_federated
zsh: killed python3
To compile tensorflow_federated. I had to modify the .bazelrc file.
To install the pip package I had to run with --no-deps option because the installed tensorflow package is named tensorflow-macos not tensorflow.
pip3 install --no-deps /tmp/tensorflow_federated/tensorflow_federated-0.27.0-py2.py3-none-any.whl
Since I ran with no-deps I had to manually install the following pip packages
- tensorflow_privacy
- attrs
- tensorflow_model_optimization
- pyfarmhash
Anyone have tips on how to debug the crashing? I tried to enable an application core dump but so far haven't had any success.
tensorflow-federated==0.38.0 doens't work with tensorflow-macos==2.10.0, the arm package for tensorflow-compression, tensorflow_model_optimzation are not available. Wasn't even able to build tensorflow-compression manually with bazel.
I've managed to get tff to work on my M1 Pro. It's worth noting that I'm only testing briefly, complex jobs don't know the effect. As @yingding said tensorflow-federated==0.38.0 needs the support of tensorflow-compression but arm does not have this. So we need to install the version of tensorflow-federated=<0.34.0. When installing tensorflow-federated, it asks for tensorflow, and Apple silicon installs tensorflow-macos, so it can't find it. My method is similar to @jlewi, using --no-deps. But this way, other dependencies need to be installed step by step. I'll post the steps I completed below.
conda create --name experiment python=3.9 conda activate experiment conda install -c apple tensorflow-deps pip install tensorflow-2.9.1-cp39-cp39-macosx_11_0_arm64.whl pip install tensorflow_federated-0.34.0-py2.py3-none-any.whl --no-dependencies pip install attrs==21.4 pip install dm-tree==0.1.7 farmhashpy==0.4.0 portpicker==1.5 semantic-version==2.6 tensorflow-model-optimization==0.7.3 tensorflow-privacy==0.8.4 tqdm==4.64 cachetools==3.1 grpcio==1.46 --no-dependencies pip install dp-accounting==0.1.2 pip install numpy==1.23.4 pip install matplotlib==3.3.4 pandas==1.1.4 scikit-learn==1.0.2 tensorflow-datasets==4.5.2 tensorflow-probability==0.15 pip install jax==0.3.13 jaxlib==0.3.10 pip install tensorflow-metal==0.5
If you use the 0.6 version of tensorflow-metal, there will be a Bus error. I am currently just getting it to work. Stability is not known.
@Jiahui-Bai thanks for posting all the dependencies using --no-deps. Can you tell me which tensorflow-metal version did you use? 0.5.1, 0.5.0 or 0.4.0.
@Jiahui-Bai thanks for posting all the dependencies using
--no-deps. Can you tell me which tensorflow-metal version did you use? 0.5.1, 0.5.0 or 0.4.0.
I am using 0.5.0 now. I tried 0.6.0 and it doesn't work properly in tff. I have not tried other versions.
Inspired by @Jiahui-Bai, I got the following sequence with homebrew and pip working, I also upgraded to the newest package as much as I can.
# create a VENV with python3.9 names ~/VENV/tff3.9
brew install [email protected]
/opt/homebrew/bin/python3.9 -m venv ~/VENV/tff3.9
source ~/VENV/tff3.9/bin/activate
python3 -m pip install --upgrade pip
# install tensorflow-macos, metal, tff
# clear all packages (optional)
python3 -m pip freeze | xargs pip uninstall -y
python3 -m pip install tensorflow-macos==2.9.2 --no-cache-dir
python3 -m pip install attrs==21.4.0 dp-accounting==0.1.2 matplotlib==3.6.2 pandas==1.5.1 scikit-learn==1.1.3 tensorflow-datasets==4.5.2 tensorflow-probability==0.15 --no-cache-dir
python3 -m pip install farmhashpy==0.4.0 portpicker==1.5 semantic-version==2.6 tensorflow-model-optimization==0.7.3 tensorflow-privacy==0.8.4 --no-dependencies --no-cache-dir
# can not use tensorflow-metal==0.6.0 got a bus error
python3 -m pip install jax==0.3.24 jaxlib==0.3.24 tensorflow-metal==0.5.1 numpy==1.23.4 --no-cache-dir
python3 -m pip install tensorflow-federated==0.34.0 --no-deps --no-cache-dir
With a modified tff example (https://github.com/yingding/tf-every-day/blob/main/04IntelMetal/tff_gpu.py), I got:
2.9.2
Metal device set to: Apple M1 Max
systemMemory: 64.00 GB
maxCacheSize: 24.00 GB
2022-11-05 21:18:29.989492: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
13.897557
14.327111
14.385421
14.352641
14.37589
executed in 5.97835111618042 secs
Everything seems fine sofar.
Inspired by @Jiahui-Bai, I got the following sequence with
homebrewandpipworking, I also upgraded to the newest package as much as I can.# create a VENV with python3.9 names ~/VENV/tff3.9 brew install [email protected] /opt/homebrew/bin/python3.9 -m venv ~/VENV/tff3.9 source ~/VENV/tff3.9/bin/activate python3 -m pip install --upgrade pip # install tensorflow-macos, metal, tff # clear all packages (optional) python3 -m pip freeze | xargs pip uninstall -y python3 -m pip install tensorflow-macos==2.9.2 --no-cache-dir python3 -m pip install attrs==21.4.0 dp-accounting==0.1.2 matplotlib==3.6.2 pandas==1.5.1 scikit-learn==1.1.3 tensorflow-datasets==4.5.2 tensorflow-probability==0.15 --no-cache-dir python3 -m pip install farmhashpy==0.4.0 portpicker==1.5 semantic-version==2.6 tensorflow-model-optimization==0.7.3 tensorflow-privacy==0.8.4 --no-dependencies --no-cache-dir # can not use tensorflow-metal==0.6.0 got a bus error python3 -m pip install jax==0.3.24 jaxlib==0.3.24 tensorflow-metal==0.5.1 numpy==1.23.4 --no-cache-dir python3 -m pip install tensorflow-federated==0.34.0 --no-deps --no-cache-dirWith a modified tff example (https://github.com/yingding/tf-every-day/blob/main/04IntelMetal/tff_gpu.py), I got:
2.9.2 Metal device set to: Apple M1 Max systemMemory: 64.00 GB maxCacheSize: 24.00 GB 2022-11-05 21:18:29.989492: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz 13.897557 14.327111 14.385421 14.352641 14.37589 executed in 5.97835111618042 secsEverything seems fine sofar.
It works fine on my computer. 👍
Inspired by @Jiahui-Bai, I got the following sequence with
homebrewandpipworking, I also upgraded to the newest package as much as I can.# create a VENV with python3.9 names ~/VENV/tff3.9 brew install [email protected] /opt/homebrew/bin/python3.9 -m venv ~/VENV/tff3.9 source ~/VENV/tff3.9/bin/activate python3 -m pip install --upgrade pip # install tensorflow-macos, metal, tff # clear all packages (optional) python3 -m pip freeze | xargs pip uninstall -y python3 -m pip install tensorflow-macos==2.9.2 --no-cache-dir python3 -m pip install attrs==21.4.0 dp-accounting==0.1.2 matplotlib==3.6.2 pandas==1.5.1 scikit-learn==1.1.3 tensorflow-datasets==4.5.2 tensorflow-probability==0.15 --no-cache-dir python3 -m pip install farmhashpy==0.4.0 portpicker==1.5 semantic-version==2.6 tensorflow-model-optimization==0.7.3 tensorflow-privacy==0.8.4 --no-dependencies --no-cache-dir # can not use tensorflow-metal==0.6.0 got a bus error python3 -m pip install jax==0.3.24 jaxlib==0.3.24 tensorflow-metal==0.5.1 numpy==1.23.4 --no-cache-dir python3 -m pip install tensorflow-federated==0.34.0 --no-deps --no-cache-dirWith a modified tff example (https://github.com/yingding/tf-every-day/blob/main/04IntelMetal/tff_gpu.py), I got:
2.9.2 Metal device set to: Apple M1 Max systemMemory: 64.00 GB maxCacheSize: 24.00 GB 2022-11-05 21:18:29.989492: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz 13.897557 14.327111 14.385421 14.352641 14.37589 executed in 5.97835111618042 secsEverything seems fine sofar.
Just wanted to follow up on this thread to say that it appears this solution is still working! Thanks.
Hey there @yingding @SHVleV9CYWkK , I have tried to follow your solution, but I am not able to install as
python3 -m pip install attrs==21.4.0 dp-accounting==0.1.2 matplotlib==3.6.2 pandas==1.5.1 scikit-learn==1.1.3 tensorflow-datasets==4.5.2 tensorflow-probability==0.15 --no-cache-dir
python3 -m pip install jax==0.3.24 jaxlib==0.3.24 tensorflow-metal==0.5.1 numpy==1.23.4 --no-cache-dir
provides multiple deps issues, that I am unable to isolate and fix as upgrading one affects the deps of another. Are there any other possible issues for Mac M1?
Hey guys, I am not working around this case. So, I am going to close it as we have found the workaround and there is no response for years.