handle partially quantized models

[davidkoski]

• fix for #53 #71 #69 #74
• in order to test the models
  • I added a default prompt of an appropriate form
  • while working on the model configuration also added additional stop tokens (#74)
• fixed the repetitionPenalty code (#71)

Note: this isn't directly usable as it requires https://github.com/ml-explore/mlx-swift/pull/73 (I was using the local checkout for development). This will need an update to the mlx version after 73 is merged.

[davidkoski]

note: the CI tests are failing because this depends on the mlx-swift change that needs to merge first

[solume]

tested it with mlx-swift fork mlx-swift fork, this breaks for me with Libraries/LLM/Load.swift:62:13 Cannot find 'quantize' in scope and Libraries/LLM/Qwen2.swift:198:37 Value of type 'Embedding' has no member 'asLinear'

[davidkoski]

tested it with mlx-swift fork mlx-swift fork, this breaks for me with Libraries/LLM/Load.swift:62:13 Cannot find 'quantize' in scope and Libraries/LLM/Qwen2.swift:198:37 Value of type 'Embedding' has no member 'asLinear'

That sounds like the hookup with that other branch didn't work. See:

• https://github.com/ml-explore/mlx-swift/pull/73/files#diff-9edad027266f523cf845c47a29ad9d65622849922650d5333910c563987794e0R43

[solume]

was able to fix dependencies but getting runtime error: libc++abi: terminating due to uncaught exception of type std::invalid_argument: [matmul] Last dimension of first input with shape (1,42,1280) must match second to last dimension of second input with shape (160,32000).