unsloth
unsloth copied to clipboard
unslothai
Apple Silicon Support
The more upvotes this gets will equal to the priority!! If it's a big must - I'll try my best to implement it!
@ChristianWeyer Oh my 30!! Ok ok this definitely is a very good indication that people want Apple + MLX support :)) I will have to readjust my engineering roadmap :)
Yeee I can see this is super popular - sadly I thought more about it - itll be a bit more complicated than I though + until I get a Mac (I use Linux + Windows!!!) it'll be a bit of an issue for me to test training runs :( I thought maybe I could just take MLX and sprinkle some Unsloth magic, but sadly that's not the case :(
@danielhanchen I have an M3 Max Macbook Pro with 128GB Unified RAM. Happy to help you port this great project to Apple Silicon 😃 If it's any helpful, I can spin up a VM and give you SSH access. IIRC macOS VM's on AS have direct GPU access and can run Metal with near native performance.
I seriously think that if you made a crowdfunding and posted it here and on LLAMA subreddit we could collectively get you the funds to buy a good Apple Silicon machine. Although I am wondering if it's doable at this stage from a software perspective. I am not sure how much you would have to rely on frameworks like MLX / Metal, but I do know their development is very active for the last couple of months, so it might become viable soon.
Please support Apple Mac silicon M2、M3 series chips！！ It would be perfect！！！ pls！
Just to throw this out there: According to Apple (which notes MPS support with PyTorch version 1.12) and PyTorch (a press release dated May 2022), Apple Silicon Metal Performance Shaders (MPS) support has been standard in PyTorch for a while.
In fact, the standard pip install torch installation will do everything needed (i.e. nothing special) to enable PyTorch GPU acceleration on Apple Silicon. I personally stopped installing the nightly download months ago, once I realized that I was putting way more effort into making sure my version was up-to-date than was necessary.
To note, pip install torch installs version 2.3.0 as of this writing.
At this point it almost seems like more effort to not to support Metal. I'm sure there's complexities and reasons for it that I'm not aware of, but I wonder if there's old practices that pervade the LLM space that need to be mixed up?
Oh interesting if Pytorch already has been on MPS, hmm I'll see what I can do! (Once I get a Mac :))
Here's an interesting article arguing that any speedup from MLX over PyTorch's MPS is due to a small optimization mistake that could be easily fixed. The article is from a few months ago so that optimization issue might even be fixed by now: https://medium.com/@koypish/mps-or-mlx-for-domestic-ai-the-answer-will-surprise-you-df4b111de8a0
It does seem like PyTorch's MPS implementation is the move 👀, I agree with everyone above that Unsloth on Apple Silicon would be heavenly 🙏
@danielhanchen Another Mac user, chugging along on my M1. So don't forget us when people talk about newer chips, please. Now as to what you need to do this work: what about renting a Mac? Here's a simple google search result: https://www.google.com/search?client=safari&rls=en&q=rent+apple+silicon&ie=UTF-8&oe=UTF-8 I note one of them is even pay as you go. You could probably raise enough money of a short term rental just from the people on this thread. Check it out, give us a budget, and let the magic happen.
Sorry on the delay - just relocated to SF!
It isn't the issue of renting - it's more I'm not used to it :( I'll try getting one in the following weeks :)
Hey, I wanted to throw out a lesser stepping-stone sort of request -- can there at a minimum be mac wheels with stubs?
For example, my team develops on macs but runs experiments and deploys models onto linux, with cuda GPUs. However, developing on mac when the unsloth package cannot be installed in a local environment means that e.g. IDEs cannot tab-complete to suggest class or method names, linters/checkers cannot flag when you misspelled a keyword argument, etc.
To me it's acceptable that to run an experiment, I need to provision a GPU instance. Even if it 'can' work on my laptop, it's likely faster and worthwhile for me to run it on a more powerful instance. But running a GPU instance just be able to install the package and discover (as a runtime error) that I accidentally passed warm_up_ratio=0.1 in UnslothTrainingArguments when actually it needs to be warmup_ratio ... seems like a waste of both human time and machine time. That's the kind of thing that a linter could have caught locally, if only I were able to install (but not run) unsloth in my mac development environment.
Oh interesting - ie just unsloth as a pure python package on Mac, but temporarily disable the fast code paths?
Yeah, so far as I'm concerned, if it throws "NotImplemented" errors, but has the class/method signatures, that would be a meaningful step forward from my point of view. Roughly, developers and CI environments would be able to set up an environment that lets them check/lint even if they can't run the full functionality. If that gives you an incremental path to eventually provide that full behavior, that's great!
On Fri, Jul 5, 2024, 20:26 Daniel Han @.***> wrote:
Oh interesting - ie just unsloth as a pure python package on Mac, but temporarily disable the fast code paths?
— Reply to this email directly, view it on GitHub https://github.com/unslothai/unsloth/issues/4#issuecomment-2211612297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABD6MWW37OFEV5DHPFBE23ZK5PV5AVCNFSM6AAAAABADU7SJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGYYTEMRZG4 . You are receiving this because you commented.Message ID: @.***>
I just need to develop/test chat templates w/o burning through GPU hours. Any pointers on how you develop them without incurring the same costs?
The more upvotes this gets will equal to the priority!! If it's a big must - I'll try my best to implement it!
125 now.. Any news on progress
I guess the main problem right now is Apple's ability to support Triton and other Unsloth reliance's. Making Triton work for Apple GPUs is a lot of work and Apple hasn't reached out as far as we're aware of.
It is however possible for us to support Apple but with less optimizations (basically with no Triton optimizations) than what is currently available for NVIDIA GPUs. Is that what you guys want?
Hey, I'm jumping in here. I discovered unsloth recently, and I am quite enthusiastic about the project. It seems very promising. On my side, I don't want any optimization when running on Apple silicon. I just want to be able to write code and run it to debug. The proper computation would run on a server where unsloth and its optimization can be installed.
