triton-inference-server
Support Mac installation
It would be great to be able to install pytriton on Macs for ease-of-development. Even with the lack of CUDA support for Macs, being able to develop using only the CPU would be a real time saver.
As a far reaching stretch goal, being able to use mps on Macs would also be great.
As of now pip install nvidia-pytriton throws an error being unable to install cuda-python on my machine.
My Machine:
- Apple M2 Max
- 32GB Memory
- Year - 2023
- Python - 3.11.6
As far as I understand, that update should come from triton-inference-server, but since it's supposed to run on an NDVIDIA GPU I don't think it'll happen.
What you can do is run on a docker emulated image running docker image with flag --platform linux/amd64. It will run on CPU and slower but you'll be able to debug.
Hope that helps
The limitation we have is installation on Linux machine as this is how the binary and libraries are prepared. As @sricke mentioned you may run Ubuntu:22.04 system using Docker and install recently published wheel with aarch64 support which you can find here: https://github.com/triton-inference-server/pytriton/releases/download/v0.4.1/nvidia_pytriton-0.4.1-py3-none-manylinux_2_35_aarch64.whl (publishing to pypi is in progress).
Thanks for the quick fix, that's what I'll start with.
Is it possible to decouple pytriton from the cuda-python library?
The cuda-python is required by tritonclient package which is PyTriton dependency. As for now we are not able to remove it, but if you would run installation inside the Docker container with Ubuntu 22.04 there should be no issue despite the CUDA is not available in the system.
@jkosek It seems cuda-python is optional for tritonclient, is it possible to make it optional in pytriton?
@martin-liu you are correct. We are going to fix dependencies on our side to avoid cuda dependency at the moment. Thanks!
@jkosek, fantastic! I'm truly appreciative of your efforts to resolve this issue. This adjustment will significantly enhance local development experience on Mac. Thank you!
@martin-liu the change has been applied on the main branch and will be available with next wheel release.
@jkosek Seems there are no wheels available that support macOS. Are there any specific issues or blockers preventing the creation of macOS wheels?
Hey @martin-liu . At the moment we are not able to provide a macOS wheel. The PyTriton contains a Triton binaries which are bounded to the Linux OS and specific version of GLibc. In such case the only option we can suggest is to use Docker image, ex. Ubuntu, on macOS.