[Feature] Support unit32 in save_gguf

[mzbac]

While working on converting the quant MLX model to GGUF format, I noticed that we do not support uint32 in save_gguf. This makes it difficult to convert the quant mode given that the weight of quant models are in uint32 format. I am wondering if there is any chance we would support uint32 format in the save_gguf method?

the code to replicate the issue:

import mlx.core as mx


weights = {"test": (mx.ones(shape=(2,), dtype=mx.uint32))}


mx.save_gguf("mlx_model.gguf", weights)

[awni]

So I think you're goal is to export weights, scales, and biases to GGUF without dequantizing so you can load them natively in llama.cpp right?

It is not quite so simple as enabling a uint32 format (which would actually be the wrong thing). GGUF stores quantized weights in a completely different format, and it is packed. So all three tensors (weights, scales, biases) would get packed into e.g. a q4_0 tensor. Doing that properly is a bit messy hence right now we don't really support exporting quantized models into GGUF.

[awni]

I will leave this open as an enhancement to help prioritize when we can get to it. For the very short-term your best bet is exporting to fp16 (either safetensors or gguf) and then quantizing.

[mzbac]

I see, I thought that since we can load a quantized gguf format model without de-quantization in mlx, maybe we can directly convert mlx quant to gguf, which would be more beneficial than converting it to f16 gguf format and quantize it

[dc-dc-dc]

Quick note, not sure if uint32 is a supported type in gguf, only see signed ints on the list