[BUG]: For Mac M1, fresh `pip install zenml` in python 3.9.13 has dependency clashes that prevent install

[j-hartshorn]

Contact Details [Optional]

No response

System Information

Python 3.9.13 (main, Jun 6 2022, 18:15:49) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin macOS Monterey 12.2.1 (21D62)

What happened?

Creating a brand new python 3.9 environment (with pyenv) and attempting to install zenml with pip fails due to dependency clashes. I would expect this to just work.

Reproduction steps

1. Install python pyenv install 3.9.13
2. Set local version pyenv local 3.9.13
3. Create and activate virtual environment python -m venv venv and source venv/bin/activate
4. attempt to install zenml pip install zenml ...

Relevant log output

$ pip install zenml
Collecting zenml
  Using cached zenml-0.8.1-py3-none-any.whl (623 kB)
Collecting fastapi<0.76.0,>=0.75.0
  Using cached fastapi-0.75.2-py3-none-any.whl (54 kB)
Collecting nbconvert==6.4.4
  Using cached nbconvert-6.4.4-py3-none-any.whl (561 kB)
Collecting gitpython<4.0.0,>=3.1.18
  Using cached GitPython-3.1.27-py3-none-any.whl (181 kB)
Collecting pydantic<2.0.0,>=1.9.0
  Using cached pydantic-1.9.1-cp39-cp39-macosx_11_0_arm64.whl (2.5 MB)
Collecting python-dateutil<3.0.0,>=2.8.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting rich[jupyter]<13.0.0,>=12.0.0
  Using cached rich-12.4.4-py3-none-any.whl (232 kB)
Collecting httplib2<0.20,>=0.19.1
  Using cached httplib2-0.19.1-py3-none-any.whl (95 kB)
Collecting analytics-python<2.0.0,>=1.4.0
  Using cached analytics_python-1.4.0-py2.py3-none-any.whl (15 kB)
Collecting uvicorn[standard]<0.18.0,>=0.17.5
  Using cached uvicorn-0.17.6-py3-none-any.whl (53 kB)
Collecting pandas<2.0.0,>=1.1.5
  Using cached pandas-1.4.2-cp39-cp39-macosx_11_0_arm64.whl (10.1 MB)
Collecting distro<2.0.0,>=1.6.0
  Using cached distro-1.7.0-py3-none-any.whl (20 kB)
Collecting click<9.0.0,>=8.0.1
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting apache-beam<3.0.0,>=2.30.0
  Using cached apache-beam-2.39.0.zip (2.8 MB)
  Preparing metadata (setup.py) ... done
Collecting markupsafe==1.1.1
  Using cached MarkupSafe-1.1.1.tar.gz (19 kB)
  Preparing metadata (setup.py) ... done
Collecting ml-pipelines-sdk==1.8.0rc1
  Using cached ml_pipelines_sdk-1.8.0rc1-py3-none-any.whl (1.3 MB)
Collecting semver<3.0.0,>=2.13.0
  Using cached semver-2.13.0-py2.py3-none-any.whl (12 kB)
Collecting sqlmodel<0.1.0,>=0.0.6
  Using cached sqlmodel-0.0.6-py3-none-any.whl (21 kB)
Collecting pyparsing<3,>=2.4.0
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pyyaml<6.0.0,>=5.4.1
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting click-params<0.4.0,>=0.3.0
  Using cached click_params-0.3.0-py3-none-any.whl (12 kB)
Collecting portpicker<2,>=1.3.1
  Using cached portpicker-1.5.0-py3-none-any.whl (14 kB)
INFO: pip is looking at multiple versions of markupsafe to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of zenml to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement ml-metadata<1.9.0,>=1.8.0 (from ml-pipelines-sdk) (from versions: 0.12.0.dev0, 0.13.0.dev0, 0.13.1.dev0)
ERROR: No matching distribution found for ml-metadata<1.9.0,>=1.8.0

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[htahir1]

@j-hartshorn I am unable to replicate this issue. Our github actions tests run on Python 3.9 and are able to install ZenML fine. I also verified that ml-metadata==1.8.0 has Python 3.9 support (https://pypi.org/project/ml-metadata/1.8.0/) .

Are you sure you are using the right environment when you do pip install zenml ?

[htahir1]

Ah I see you are using macOS Monterey 12.2.1 (21D62) -> Perhaps the issue is you are on a Mac M1 processor? For the M1's, we have a known limitation and problems installing zenml. See this guide for details and please let us know if it worked!

[alfredodeza]

@htahir1 I just hit the same problem, and although I think it is useful that there is a documented workaround, but it isn't obvious for a user that this is in fact a problem due to an incompatibility with M1s.

Since this is problematic today, I would recommend erroring on ARM64 architectures with a meaningful message. I see that ZenML doesn't use setup.py which can handle this situation, so I can't say if this is possible with the tooling.

Another thing that would be useful is to keep this issue open to spread awareness. In my particular case, the linked guide uses the one packaging framework/tooling that I don't use, so I'll keep looking for other ways to try this outt.

[j-hartshorn]

Perhaps the issue is you are on a Mac M1 processor Ahh yes, I think that's probably the issue. Thanks for spending some time on this issue!

[htahir1]

Great @j-hartshorn. Can you verify if the workaround fixed?

@alfredodeza You're right but then lets change the issue name to reflect it closely

This is also related #93

[alfredodeza]

@htahir1 is the plan to steer away from the Tensorflow dependency as noted in https://github.com/zenml-io/zenml/issues/93#issuecomment-887324971 ?

[j-hartshorn]

@htahir1 I'm reluctant to install an entirely new python toolchain to handle this particular case, it seems like a severe workaround. We have alternate linux environments we can use zenml that work in the mean time and I'll keep an eye out for potential fixes in the future.

[htahir1]

@alfredodeza We do not have a tensorflow dependency. If you mean the dependency with ml-pipelines-sdk, then that is definitely something on the roadmap to do but I cant give a clear timeline for it.

@j-hartshorn Sorry about that! The Mac M1 experience is really painful. Even within the core maintainer team, we have a few people using it like @safoinme and @schustmi and we know the pain for sure.

If there are any suggestions or helpful advice, we're happy to take them on board. In the meanwhile, ill keep this issue open and hope that upstream dependencies release support soon!

[Karamya]

Any updates on this issue. Tried completely new python environments with different python versions (3.7.x, 3,8.x and 3.9.x) and it is impossible to install zenml on Mac M1. Tried different versions of Tensorflow with and without M1 processor support. Will wait for the updates before exploring zenml. Thanks

[htahir1]

@Karamya We have now a dedicated guide just for this: https://docs.zenml.io/getting-started/installation/m1-mac-installation

[schustmi]

@alfredodeza @j-hartshorn @Karamya Starting with zenml==0.30.0 (released yesterday) we now natively support M1 macs!