CodeTF icon indicating copy to clipboard operation
CodeTF copied to clipboard

Published 20 hours ago • verified publisher icon salesforce

MPS support

Open tmm1 opened this issue 2 years ago • 1 comments
trafficstars

It would be nice to support Metal Performance Shaders backend for pytorch on macOS

the main changes for inference seem to be:

  • load_in_8bit must be False
  • device_map set to {"":"mps"}

see:

https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L154-L157

https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L175

cc @Birch-san are there any other good docs or resources for adding mps support?

tmm1 avatar Jun 28 '23 17:06 tmm1

this is great, can you help to propose any pull request that i can merge?

bdqnghi avatar Jul 04 '23 03:07 bdqnghi