Falled to load Qwen3 model locally

[Linorman]

This the function that gives error: MLXLMCommon.ModelFactoryError error 0. I call this function with type: 'qwen3'

func loadModelContext(modelConfiguration: ModelConfiguration, type: String) async throws -> ModelContext {
        let configurationURL = modelConfiguration.modelDirectory().appendingPathComponent("config.json")
        let baseConfig = try JSONDecoder().decode(BaseConfiguration.self, from: Data(contentsOf: configurationURL))
        let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)
        try loadWeights(modelDirectory: modelConfiguration.modelDirectory(), model: model, quantization: baseConfig.quantization)
        let tokenizer = try await loadTokenizer(configuration: modelConfiguration, hub: HubApi(downloadBase: modelConfiguration.modelDirectory()))
        let processor = LLMUserInputProcessor(tokenizer: tokenizer, configuration: modelConfiguration)
        return ModelContext(configuration: modelConfiguration, model: model, processor: processor, tokenizer: tokenizer)
    }

When I use Qwen2 model and change type to "qwen2", everything works well, so I can't figure out what's going on, and I am sure I use the latest version of mlx-examples (main branch).

[davidkoski]

That looks ok to me. I think you are seeing this error:

public enum ModelFactoryError: LocalizedError {
    case unsupportedModelType(String)

I presume from this line:

        let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

and in current main there are qwen3 types registered:

            "qwen3": create(Qwen3Configuration.self, Qwen3Model.init),
            "qwen3_moe": create(Qwen3MoEConfiguration.self, Qwen3MoEModel.init),

Two suggestions:

• can you use llm-tool or one of the example apps to load the model?
• what model (hugging face id) are you using?

[davidkoski]

Possibly #291 (see #293 )

[Linorman]

That looks ok to me. I think you are seeing this error:

public enum ModelFactoryError: LocalizedError { case unsupportedModelType(String) I presume from this line:

    let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

and in current main there are qwen3 types registered:

        "qwen3": create(Qwen3Configuration.self, Qwen3Model.init),
        "qwen3_moe": create(Qwen3MoEConfiguration.self, Qwen3MoEModel.init),

Two suggestions:

• can you use llm-tool or one of the example apps to load the model?
• what model (hugging face id) are you using?

Thanks. I am using a finetuned model based on Qwen3-1.7B and I haven't uploaded it to huggingface by now. And I do use a quant model which is quanted with mlx framework using python. I load this model successfully using python and mlx.

[Linorman]

There is no huggingface id so I can't test using llm-tool

[davidkoski]

It should work if you do --model /path/to/model/directory

[Linorman]

It should work if you do --model /path/to/model/directory

Thanks, I tried and it loaded successfully. So there is something wrong with other codes?

[davidkoski]

Nothing obvious, but I would look at this part:

        let configurationURL = modelConfiguration.modelDirectory().appendingPathComponent("config.json")
        let baseConfig = try JSONDecoder().decode(BaseConfiguration.self, from: Data(contentsOf: configurationURL))
        let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

Typically you get the type from baseConfig.modelType. You might try printing the value being passed in vs what is in baseConfig. I suspect that type is simply wrong.

[Linorman]

Nothing obvious, but I would look at this part:

    let configurationURL = modelConfiguration.modelDirectory().appendingPathComponent("config.json")
    let baseConfig = try JSONDecoder().decode(BaseConfiguration.self, from: Data(contentsOf: configurationURL))
    let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

Typically you get the type from baseConfig.modelType. You might try printing the value being passed in vs what is in baseConfig. I suspect that type is simply wrong.

It's a little bit tricky. I changed type to baseConfig.modelType instead but it still gives the same error. I checked the dependencies again and I am sure I use the correct version.

Do I have other ways to load model locally?

[davidkoski]

OK, what is the value of the type/modelType? That needs to match this list:

• https://github.com/ml-explore/mlx-swift-examples/blob/main/Libraries/MLXLLM/LLMModelFactory.swift#L31

[Linorman]

OK, what is the value of the type/modelType? That needs to match this list:

• https://github.com/ml-explore/mlx-swift-examples/blob/main/Libraries/MLXLLM/LLMModelFactory.swift#L31

qwen3, it's a finetuned model based on Qwen3-1.7B, this is the config file:

{
  "architectures": [
    "Qwen3ForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 2048,
  "initializer_range": 0.02,
  "intermediate_size": 6144,
  "max_position_embeddings": 40960,
  "max_window_layers": 28,
  "model_type": "qwen3",
  "num_attention_heads": 16,
  "num_hidden_layers": 28,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000,
  "sliding_window": null,
  "tie_word_embeddings": true,
  "torch_dtype": "float16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 151936
}

[Linorman]

Ok, I find something strange, in the latest version of mlx-swift-examples, there is no asMessage() function in UserInput, and I still call this function in my codes but build without errors, so maybe here is the problem.

[davidkoski]

yes, that sounds like you are using an older checkout / tag, but from your screenshot you are up-to-date. If you cmd-click on that symbol, see what source it jumps to for the definition.

Do I have other ways to load model locally?

you just need to create a ModelConfiguration pointing to a local directory:

https://swiftpackageindex.com/ml-explore/mlx-swift-examples/main/documentation/mlxlmcommon/modelconfiguration/init(directory:tokenizerid:overridetokenizer:defaultprompt:extraeostokens:)

the you can use all the standard loading code.

[Linorman]

yes, that sounds like you are using an older checkout / tag, but from your screenshot you are up-to-date. If you cmd-click on that symbol, see what source it jumps to for the definition.

Do I have other ways to load model locally?

you just need to create a ModelConfiguration pointing to a local directory:

https://swiftpackageindex.com/ml-explore/mlx-swift-examples/main/documentation/mlxlmcommon/modelconfiguration/init(directory:tokenizerid:overridetokenizer:defaultprompt:extraeostokens:)

the you can use all the standard loading code.

Thanks for your help. By now I still haven't solved this problem. I use cmd-click and it jumps to the right definition so it seems the version is correct. But when I build it still uses the old version. I believe there is something wrong with Xcode or other things and I will try to get help from xcode community. Thanks again for your time and help.

[davidkoski]

OK, perhaps we can catch it failing and see something that way.

This method in ModelTypeRegistry is what throws the error:

    public func createModel(configuration: URL, modelType: String) throws -> LanguageModel {
        let creator = lock.withLock {
            creators[modelType]
        }
        guard let creator else {
            throw ModelFactoryError.unsupportedModelType(modelType)
        }
        return try creator(configuration)
    }

So you could try setting a breakpoint on the throw of that error:

Then you can look at self or creators and see if they look unsual. You can also look at the path in Xcode and see if that isn't coming from the place that you expect.

[davidkoski]

My guess is that there are somehow two copies of mlx-swift or mlx-libraries (the package inside of mlx-swift-examples). One is current and one is older and the older one is getting called.

You could see something like this especially if you have two frameworks that link MLX or a framework and an application that link MLX. Since it is a swiftpm library it could get linked (and copied) into each of the targets. Maybe something like that.

[Linorman]

OK, perhaps we can catch it failing and see something that way.

This method in ModelTypeRegistry is what throws the error:

public func createModel(configuration: URL, modelType: String) throws -> LanguageModel {
    let creator = lock.withLock {
        creators[modelType]
    }
    guard let creator else {
        throw ModelFactoryError.unsupportedModelType(modelType)
    }
    return try creator(configuration)
}

So you could try setting a breakpoint on the throw of that error:

Then you can look at `self` or `creators` and see if they look unsual. You can also look at the path in Xcode and see if that isn't coming from the place that you expect.

Ok, thanks. But actually my project is a Flutter project and use platform channel to call swift code, so I can't use xcode to debug which means I can't set any breakpoint in swift codes. I will try to print more info when running the project.

[Linorman]

My guess is that there are somehow two copies of mlx-swift or mlx-libraries (the package inside of mlx-swift-examples). One is current and one is older and the older one is getting called.

You could see something like this especially if you have two frameworks that link MLX or a framework and an application that link MLX. Since it is a swiftpm library it could get linked (and copied) into each of the targets. Maybe something like that.

Ok, I will check this out later.