[bugfix] Fix logic bug to make `device` argument be respected when passed in

[ms337]

Why

Currently if you pass in the device arg to from_pretrained, there is a logic bug which makes it get resolved to something else unnecessarily based on the device available.

For example, to reproduce, pass in device="cpu" on an Mac and it can resolved to "mps".

Turns out its just a bug in terms of logic in how the device is being set .

What Changed

Fixed in this PR.