Blaizzy
Add `AutoModel` support
About this feature
I think it would be highly useful to add AutoModel support to the current library. Something that goes like
from mlx_vlm import AutoModel
model = AutoModel.from_pretrained("hf_repo_or_path")
Auxiliary info
I was able to get this feature running on pxia by checking the model_info first (specifically the repo tag which is stored in the readme file) then calling the appropriate model architecture
I'm not currently in possession of any mac resources so i can't work on this, but feel free to reach out if you have any further questions.
additionally if you want to make the file download more optimized i can share this interesting resource HF's PyTorchModelHubMixin as well
Hey @not-lain
Sounds like a great idea!
But I have a question: What is the advantage of using AutoModel, compared to Load(the current approach)?
Hi @Blaizzy The reason why i'm requesting this is because for some applications we only need just the model to be used as a backbone for bigger models, in these cases we don't really need a tokenizer/preprocessor.
Hey @not-lain
I'm highly considering an AutoModel approach but for now you can use load_model or import the model class as we do here: https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/smolvlm/smolvlm.py