tabby
tabby copied to clipboard
TabbyML
M1 Mac support
I have this project now working on my M1 Mac well enough that I can visit http://127.0.0.1:8081/#/default/completions_v1_completions_post and do a successful test post of the binarysearch completions example. Skimming the logs, the only clear error remaining is this:
2023-04-11 12:12:00,170 DEBG 'dagu_scheduler' stdout output:
2023/04/11 12:12:00 Failed to start DAG: fork/exec ./tabby/tools/analytic/main.sh: no such file or directory
Some notes:
- My strategy was to avoid Docker since the benefit of Metal would be lost unless I ran this natively
- I added a few instruction changes to the README.md that I used to get things set up on my local machine
- I had to replace the absolutely path of
/datawith the relative path ofdatain many instances. I don't know how to write that in a backward compatible way. - I added
torchvisionandsupervisortopyproject.toml. I experimented with some ways to make it ONLY install on the Mac to not affect the original Dockerfile behavior but couldn't get it to work. I'm a Python novice and today is the first time I even heard ofpoetryso it's probably entirely possible for someone else to fix.
I hope this helps you speed this project along! Like I said, I'm a Python novice, but I'm very interested in this project and eager to assist. Let me know what else I can do to help.
One option would be to define e.g. DATA_DIR and then use /data if it exists (eg you’re in a container) or make and use ./data if it doesn’t.
Oh I forgot one more thing: I changed to port 5001 because there's a core macos process that runs on 5000 and conflicts with this.
Thank you for submitting the PR. It provides a great reference point for the mps device support. However, we won't be able to merge it.
To achieve complete support, adding an arm docker image would be better, and we should remove all the fine-tuning stuff as it's unlikely that we'll fine-tune on a Mac client.
Nevertheless, we appreciate your contribution and will mark this PR as draft for referencing purposes.
My goal was to have performant M1 Mac support. Based on my research, it seems, this will not be possible to achieve that using Docker, which is why I opted to avoid it and execute the scripts natively. Perhaps you can an find a way!
My goal was to have performant M1 Mac support. Based on my research, it seems, this will not be possible to achieve that using Docker, which is why I opted to avoid it and execute the scripts natively. Perhaps you can an find a way!
You are correct that MPS (metal performance shader) is currently not supported in Docker, even with the arm64 Linux image. However, there are other libraries, such as GGML, that utilize the neural computing power of M1/M2. This is described in issue #66.
We will track the request running on Tabby in a non-Docker environment in issue #90