microsoft
Support for Apple MLX backend
Hi @notnotrishi, I'm a bit new to the MLX world. Do most people use the mlx_lm subpackage to interface with MLX based transformer models? https://github.com/ml-explore/mlx-examples/blob/main/llms/README.md
@Harsha-Nori yes, that's my understanding.
also, awni from mlx team is very helpful should your team has any questions
Hi, I don't know if someone pick up this but if not I would like to work on it (assuming you have time before it got implemented 😄 ).
The plan if I understand well is to propose something such as
from guidance import models, gen
# load a model (could be Transformers, LlamaCpp, VertexAI, OpenAI...)
mlx_model = models.MlxLM(path)
# append text or generations to the model
mlx_model + f'Do you want a joke or a poem? ' + gen(stop='.')
And to do so we "just" need to add a _mlx.py in the models sub package (or like transformers with a sub package) using the mlx_lm library to handle the LLM (or the https://github.com/ToluClassics/mlx-transformers mlx-transformers cause it would be easier since I would only have to "copy" the _transformers.py or even just modify the _transformers.py via good check at the right place).
Am I totally off the track or Is it the path to follow?
