David Koski

Ah yes, the quant code has no idea what to do with that (yet) -- I haven't seen this format before.

I like the look of `Tool` -- this looks straightforward for how to integrate with existing models. The tricky part (and I think this is what you are alluding to...

> Base models don't usually have a chat template. Is there something like https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/generate.py#L213C1-L214C1 in Swift Transformers? No, nothing like that: https://github.com/huggingface/swift-transformers/blob/main/Sources/Tokenizers/Tokenizer.swift#L359 Perhaps there should be? @pcuenca @maiqingqiang

Per above I added a workaround -- it will try to apply the template, but if that throws an error it will fall back to a built in template and...

We now have proper support for this as of #288

@ibrahimcetin FYI @blaizzy I think the python `mlx-vlm` has the same issue (roughly). It doesn't have the same structures but it does treat the media as a single bundle. Assuming...

That looks ok to me. I think you are seeing this error: ```swift public enum ModelFactoryError: LocalizedError { case unsupportedModelType(String) ``` I presume from this line: ```swift let model =...

Possibly #291 (see #293 )

It should work if you do `--model /path/to/model/directory`

Nothing obvious, but I would look at this part: ```swift let configurationURL = modelConfiguration.modelDirectory().appendingPathComponent("config.json") let baseConfig = try JSONDecoder().decode(BaseConfiguration.self, from: Data(contentsOf: configurationURL)) let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)...