David Koski

Results 49 comments of David Koski

When to stop in the LLMEval?

Specifically the code is looking for either the unknown token or the eos token: ```swift if t == tokenizer.unknownTokenId || t == tokenizer.eosTokenId { ``` https://github.com/ml-explore/mlx-swift-examples/blob/main/Libraries/LLM/Evaluate.swift#L199 The `didGenerate` block that...

When to stop in the LLMEval?

It may be related to this: https://github.com/huggingface/swift-transformers/issues/92 -- we are not passing in a proper prompt and the generation may be impacted. That issue is a bit terse but basically...

llm-eval: not responding to 'what is your name?' or 'what is the difference between star wars and star trek?'

That actually looks "right": ``` python -m mlx_lm.generate --model ~/Documents/huggingface/models/mlx-community/phi-2-hf-4bit-mlx --prompt 'Instruct: what is your name?. Output: ' Special tokens have been added in the vocabulary, make sure the associated...

llm-eval: not responding to 'what is your name?' or 'what is the difference between star wars and star trek?'

It looks like phi2 can't answer that prompt -- maybe it doesn't cover that info or maybe it is tool small? `mistral7B4bit` aka `mlx-community/Mistral-7B-v0.1-hf-4bit-mlx` seems to do an ok job,...

LLMEval crashing due to force unwrapping of optional in token decode for Gemma

This looks a bit like: - https://github.com/huggingface/swift-transformers/pull/52 - https://github.com/huggingface/swift-transformers/issues/51 What version (hash) of `swift-transformers` are you using?

LLMEval crashing due to force unwrapping of optional in token decode for Gemma

OK, that is the current head of main and includes that fix. Oh, interesting -- that particular prompt does reproduce it for me! Here is the token in question: ```swift...

LLMEval crashing due to force unwrapping of optional in token decode for Gemma

I had a thought about how this might be fixed -- we might be able to make something along these lines: ```swift struct CodePointString : StringProtocol { let value: String...

LLMEval: Model Configuration Registry Remains Empty, Preventing Dynamic Model Selection

You aren't missing anything, the implementation falls slightly short. Perhaps `registry` should not be public (expose via a function?) or maybe it can be done with a lazy property. Right...

Fine-tuned Qwen2 model inference error

Yeah, here are some ideas that I wrote for somebody else working on LLMs: - install the llms/mlx_lm code from https://github.com/ml-explore/mlx-examples - this gives you a working python version that...

Fine-tuned Qwen2 model inference error

If you feed the python tokens in the swift model does it produce the expected output? Yeah, it may be that there is more to the Qwen tokenizer. There must...