io
io copied to clipboard
tensorflow
arm64 support for M1
Dear TensorFlow-io team and community, are there any plans for adding support for arm64 machines? I use a M1 mac and I'm not able to install the wheel. Anyone who's reading this, do you know any alternatives that I can use? My goal is to create a mel-spectogram from a .wav file. ps: M1 cannot use Scipy at this time.
@redmlr In order to get M1 support I think we needs
- The M1 version of the tensorflow package (https://github.com/tensorflow/tensorflow/issues/44751 provides a way to build tensorflow on M1, though no release package)
- A M1 build machine (ideally with GitHub Actions) (https://github.com/actions/runner/issues/805)
I don't have access to M1 laptop yet, may take a look after I get one.
@yongtang Thank you for answering, There’s a m1 accelerated version of tensorflow in apples tensorflow repo, which is in alpha version. I’m running the latest version of this and it runs fine but tensorflow-io is a separate library which won’t install.
@redmlr The tensorflow-io package will need to be re-compiled for all functionalities to work. Though if you are only looking for functions that does not requires C++ code (only need python code) then it should be possible to just package the source tree.
Which function (or list of functions) you intend to use?
@yongtang I'm planning to use the following:
tfio.experimental.audio.trim tfio.experimental.audio.spectrogram tfio.experimental.audio.melscale tfio.experimental.audio.dbscale tfio.experimental.audio.freq_mask tfio.experimental.audio.time_mask
Thank you @yongtang ! I'm using not that long.. What does just package the source tree mean and how do I do this? I already downloaded the latest release of io and changed the desired file (init.py) as mentioned in #1302
@redmlr With PR #1302 in you can build the wheel with:
python3 setup.py -q bdist_wheel
The wheel file will be located under dist directory.
You can then install the wheel with:
python3 -m pip install --no-deps <wheel-file-build-from-last-step>
That will at least allows you to use python only functions in tensorflow-io.
Note --no-deps will ignore the dependency of tensorflow package (as the official tensorflow package is different from the apple's tensorflow package).
Note normally the wheel file needs to pass --data bazel-bin
python3 setup.py --data bazel-bin -q bdist_wheel
where --data bazel-bin indicates the location of the complied C++ kernels. However, it is not yet possible to build C++ kernels on M1 so you can skip this part.
Note also the wheel package built from the above will not have file systems support (e.g, azure/etc) as well (file system support also requires compiled C++ kernels).
Are there any updates on this front? I am interested in doing some development on TensorFlow itself, and I'm unable to install my tf-nightly wheel as there's no compatible version of tensorflow-io-gcs on pypi for arm64 macOS. I'll try setting up the io wheel from source, but being able to install straight from pypi would be great. If there's anything I can do on my end to help get this released, I'd be glad to help.
System Setup: Python 3.9 macOS 12 Apple M1 Pro processor
Steps Taken:
The below steps are how I am building TensorFlow itself, all taken from the official documentation. This stage works without issue.
$ python3.9 -m venv ./tensorflow_39_compile
$ source compile_tensorflow_39/bin/activate
$ pip install -U pip numpy wheel
$ pip install -U keras_preprocessing --no-deps
$ git clone https://github.com/tensorflow/tensorflow.git
$ cd tensorflow
$ ./configure
$ bazel build //tensorflow/tools/pip_package:build_pip_package
$ ./bazel-bin/tensorflow/tools/pip_package/build_pip_package --nightly_flag /tmp/tensorflow_pkg
Now I try to install the wheel I made, and it fails
$ cd /tmp/tensorflow_pkg
$ pip3 install -v tf_nightly-2.8.0-cp39-cp39-macosx_12_0_arm64.whl
Using pip 21.3.1 from /Users/tylerdavis/tensorflow_39_compile/lib/python3.9/site-packages/pip (python 3.9)
Processing ./tf_nightly-2.8.0-cp39-cp39-macosx_12_0_arm64.whl
Collecting keras-nightly~=2.8.0.dev
Using cached keras_nightly-2.8.0.dev2021110107-py2.py3-none-any.whl (1.4 MB)
Collecting gast<0.5.0,>=0.2.1
Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting astunparse>=1.6.0
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
ERROR: Could not find a version that satisfies the requirement tensorflow-io-gcs-filesystem>=0.21.0 (from tf-nightly) (from versions: none)
ERROR: No matching distribution found for tensorflow-io-gcs-filesystem>=0.21.0
I can confirm that the below let me install my copy of tensorflow built from HEAD:
# From root of io project
$ python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
$ cd dist
$ pip3 install tensorflow_io_gcs_filesystem-0.21.0-cp39-cp39-macosx_12_0_arm64.whl
$ cd /tmp/tensorflow_pkg
# Install h5py following the instructions in https://github.com/h5py/h5py/issues/1982
# as pip install h5py is currently broken on arm64 macs
$ pip3 install ./tf_nightly-2.8.0-cp39-cp39-macosx_12_0_arm64.whl
I too am looking to install tensorflow-io-gcs-filesystem directly from PyPI.
MacOS Monterey 12.0 Apple M1 Pro
It looks like @TylerADavis has the right commands to produce the wheel. How can we publish it to PyPI?
@matteosantama I think to make the above work out-of-the-box, it would need Github Actions to support ARM runners which is not yet on the roadmap. Also see https://github.com/actions/runner/issues/805
Cross-compilation could work in theory I guess, but the above commands aren't for cross-compiling, and I imagine it will be a little more fiddly to get it going.
No, it can't be built
(pornstar) yingshaoxo@macos io % python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
Project: tensorflow-io-gcs-filesystem
Exclude: ['tests', 'tests.*', 'tensorflow_io', 'tensorflow_io.*']
Install Requires: []
Project Rootpath: tensorflow_io_gcs_filesystem
/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
Traceback (most recent call last):
File "setup.py", line 138, in <module>
setuptools.setup(
File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/__init__.py", line 155, in setup
return distutils.core.setup(**attrs)
File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
return run_commands(dist)
File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
dist.run_commands()
File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
self.run_command(cmd)
File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 328, in run
impl_tag, abi_tag, plat_tag = self.get_tag()
File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 278, in get_tag
assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag)
AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'macosx_11_0_arm64')
No, it can't be built
(pornstar) yingshaoxo@macos io % python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem Project: tensorflow-io-gcs-filesystem Exclude: ['tests', 'tests.*', 'tensorflow_io', 'tensorflow_io.*'] Install Requires: [] Project Rootpath: tensorflow_io_gcs_filesystem /Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( Traceback (most recent call last): File "setup.py", line 138, in <module> setuptools.setup( File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/__init__.py", line 155, in setup return distutils.core.setup(**attrs) File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup return run_commands(dist) File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands dist.run_commands() File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands self.run_command(cmd) File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command cmd_obj.run() File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 328, in run impl_tag, abi_tag, plat_tag = self.get_tag() File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 278, in get_tag assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag) AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'macosx_11_0_arm64')
i got same result
No, it can't be built
i got same result
What do you guys get when you run the below?
xcrun --show-sdk-path && \
xcrun --show-sdk-version && \
sw_vers -productVersion
Ex. on my machine:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
12.1
12.1
For some of my builds I had to export MACOSX_DEPLOYMENT_TARGET="$(sw_vers -productVersion)" prior to building, but IIRC I didn't actually have to do that for TF IO...
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
12.1
12.1
same output running the above commands
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk 12.1 12.1same output running the above commands
Hmmm. so still AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'macosx_11_0_arm64')?
Is it still the same when running the below?
export MACOSX_DEPLOYMENT_TARGET="$(sw_vers -productVersion)"
python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
xcrun --show-sdk-path && \ xcrun --show-sdk-version && \ sw_vers -productVersion
I just built again without issue by doing the following from HEAD today
❯ python3.9 -m venv ~/io_v2
❯ source ~/io_v2/bin/activate
❯ pip3 install --upgrade pip
Requirement already satisfied: pip in /Users/tylerdavis/io_v2/lib/python3.9/site-packages (21.2.3)
Collecting pip
Downloading pip-22.0.3-py3-none-any.whl (2.1 MB)
|████████████████████████████████| 2.1 MB 2.0 MB/s
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.2.3
Uninstalling pip-21.2.3:
Successfully uninstalled pip-21.2.3
Successfully installed pip-22.0.3
❯ pip install wheel
Collecting wheel
Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.37.1
❯ python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
Project: tensorflow-io-gcs-filesystem
Exclude: ['tests', 'tests.*', 'tensorflow_io', 'tensorflow_io.*']
Install Requires: []
Project Rootpath: tensorflow_io_gcs_filesystem
❯ cd dist
❯ pip3 install tensorflow_io_gcs_filesystem-0.21.0-cp39-cp39-macosx_12_0_arm64.whl
Processing ./tensorflow_io_gcs_filesystem-0.21.0-cp39-cp39-macosx_12_0_arm64.whl
Installing collected packages: tensorflow-io-gcs-filesystem
Successfully installed tensorflow-io-gcs-filesystem-0.21.0
❯ xcrun --show-sdk-path && \
xcrun --show-sdk-version && \
sw_vers -productVersion
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
objc[36632]: Class AMSupportURLConnectionDelegate is implemented in both /usr/lib/libamsupport.dylib (0x1ef46f678) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1078682c8). One of the two will be used. Which one is undefined.
objc[36632]: Class AMSupportURLSession is implemented in both /usr/lib/libamsupport.dylib (0x1ef46f6c8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x107868318). One of the two will be used. Which one is undefined.
12.1
12.2
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk 12.1 12.1same output running the above commands
Hmmm. so still
AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'macosx_11_0_arm64')?Is it still the same when running the below?
export MACOSX_DEPLOYMENT_TARGET="$(sw_vers -productVersion)" python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
Hello,
I am having a similar issue installing tensorflow-io on my M1 macbook. Initially I was failing to build the wheel file just as the other two examples in this thread, however, using the commands that you suggested here I was able to do just that.
Unfortunately the installation now fails at when running:
python3 -m pip install --no-deps <wheel-file-build-from-last-step>
Which gives me the following error:
ERROR: tensorflow_io-0.24.0-cp38-cp38-macosx_10_14_arm64.whl is not a supported wheel on this platform.
I faced the same issue as Ekhao, but when I upgraded to Python 3.9 and ran it, it worked.
I was able to install tensorflow-io and tensorflow-io-gcs-filesystem but
it fails to import tensorflow_io....
tensorflow_io_gcs_filesystem works for me given the instructions above in both python 3.8 and 3.9. I can't get bazel to build tensorflow-io though on the m1, which I suppose shouldn't be surprising. This would be a great feature to have so that tfio based development would be possible on m1s.
tensorflow whl for aarch64 is now available on pypi.
https://pypi.org/project/tensorflow/2.10.0rc0/#files
I installed this tensorflow in a container on m1 mac and confirmed that tensorflow-io bazel can be built.
Buildings for AppleSilicon with MacOS are still failing due to many factors. However, being able to develop using tensorflow-io on a container without x86 emulation feels like a big step forward!
FROM python:3.9-buster
WORKDIR /root
RUN apt-get update && apt-get install wget
RUN wget https://github.com/bazelbuild/bazelisk/releases/download/v1.12.0/bazelisk-linux-arm64 -O /usr/local/bin/bazelisk && \
chmod +x /usr/local/bin/bazelisk
RUN git clone -b v0.26.0 https://github.com/tensorflow/io.git
RUN python -m pip install tensorflow==2.10.0rc0
RUN cd io && bazelisk && python tools/build/configure.py
RUN cd io && bazelisk build //tensorflow_io/... //tensorflow_io_gcs_filesystem/...
RUN cd io && \
python3 setup.py --data bazel-bin -q bdist_wheel --project tensorflow_io_gcs_filesystem && \
python3 setup.py --data bazel-bin -q bdist_wheel && \
python3 -m pip install dist/* --force-reinstall
RUN python3 -m pip install
@redmlr In order to get M1 support I think we needs
- The M1 version of the tensorflow package (TensorFlow support for Apple Silicon (M1 Chips) tensorflow#44751 provides a way to build tensorflow on M1, though no release package)
- A M1 build machine (ideally with GitHub Actions) (Add ARM + MacOS target to be able to use on self-hosted runners actions/runner#805)
I don't have access to M1 laptop yet, may take a look after I get one.
It looks like github actions has m1 build in GA now! https://github.blog/changelog/2022-08-09-github-actions-self-hosted-runners-now-support-apple-m1-hardware/
Would be great to get tf io to support mac m1 now that tensorflow supports it.
The torch project uses xcode's cross-compilation feature to build binaries for the Apple silicon version on an x86 mac. https://github.com/pytorch/pytorch/blob/ea037344e81d5645ad2e8863a9b6ccdb33f60320/.jenkins/pytorch/macos-build.sh#L36-L39 (ref. https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary)
If we try this, it seems that we can build binary for apple silicon without preparing a self-hosted runner.
This worked for me on my iMac with the latest macOS release:
pip install wheel
cd tensorflow_io
python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
cd ../project
poetry add ../tensorflow_io/dist/xxxxxxx.whl
@pshiko , I'm having issues with your Dockerfile. The last line is empty, so I removed it. Then it has lots of warnings while building, but it seems to work. When I run the container, the commands in the example on the readme of this project fail. This is the code in that example.
import tensorflow as tf
import tensorflow_io as tfio
# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
dataset_url + "train-images-idx3-ubyte.gz",
dataset_url + "train-labels-idx1-ubyte.gz",
)
And this is the error I get.
raise NotImplementedError(
NotImplementedError: unable to open file: libtensorflow_io.so, from paths: ['/usr/local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/usr/local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZN10tensorflow13GcsFileSystem12ParseGcsPathEN4absl12lts_2021110211string_viewEbPSsS4_']
If you know where is the issue or if you could help me build a container that works, I would really appreciate it. I've been trying for a while, but I'm unable to run some projects that depend on TensorFlow on my m1 Mac.
No, it can't be built
(pornstar) yingshaoxo@macos io % python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem Project: tensorflow-io-gcs-filesystem Exclude: ['tests', 'tests.*', 'tensorflow_io', 'tensorflow_io.*'] Install Requires: [] Project Rootpath: tensorflow_io_gcs_filesystem /Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( Traceback (most recent call last): File "setup.py", line 138, in <module> setuptools.setup( File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/__init__.py", line 155, in setup return distutils.core.setup(**attrs) File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup return run_commands(dist) File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands dist.run_commands() File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands self.run_command(cmd) File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command cmd_obj.run() File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 328, in run impl_tag, abi_tag, plat_tag = self.get_tag() File "/Users/yingshaoxo/miniforge3/envs/pornstar/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 278, in get_tag assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag) AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'macosx_11_0_arm64')i got same result
After some further digging, I was able to get this working on my setup with MacOS Ventura 13.0.1 (M1 Max). You have to uninstall your existing wheel version. Use the bellow commands.
pip uninstall wheel
Select yes, after the prompt to uninstall. After this, you can install wheel from its official git repository while enabling flag --use-pep517.
pip install wheel@git+http://github.com/pypa/wheel --use-pep517
You can then use the steps shown on this thread (by @TylerADavis) to install tensorflow_io. I'm providing my console output below as a reference:
(tensorflow) ---@--- io % python setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
Project: tensorflow-io-gcs-filesystem
Exclude: ['tests', 'tests.*', 'tensorflow_io', 'tensorflow_io.*']
Install Requires: []
Project Rootpath: tensorflow_io_gcs_filesystem
/Users/---/opt/miniconda3/envs/tensorflow/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
(tensorflow) ---@--- io % cd dist
(tensorflow) ---@--- dist % ls
tensorflow_io_gcs_filesystem-0.27.0-cp39-cp39-macosx_11_0_arm64.whl
(tensorflow) ---@--- dist % pip install tensorflow_io_gcs_filesystem-0.27.0-cp39-cp39-macosx_11_0_arm64.whl
Processing ./tensorflow_io_gcs_filesystem-0.27.0-cp39-cp39-macosx_11_0_arm64.whl
Installing collected packages: tensorflow-io-gcs-filesystem
Successfully installed tensorflow-io-gcs-filesystem-0.27.0
Hopefully it should solve the issue. Cheers. ;)
My solution: devcontainer and VS Code. Work in Docker give it 90% of Respources and enjoy TF without the headache
My solution: devcontainer and VS Code. Work in Docker give it 90% of Respources and enjoy TF without the headache
@muuvmuuv Can you share your dev container config? I'm running in a dev container on my M1 and still experiencing issues.
