Matt Clayton issues

Repositories
Issues
Comments

Results 2 issues of


                                            Matt Clayton

Qwen2-VL-7B-Instruct-4bit allocation crashes on larger dimension images

`Qwen2-VL-7B-Instruct-4bit` crashes on memory allocation errors on images with larger dimensions. My machine: Apple M3 Pro, 36 GB RAM Error production below is with an image of dimensions: `1978 × ...

Poor Speculative Decoding Performance on M2 Ultra

Speculative decoding does not seem to improve generation speed as expected on M2 Ultra Mac Studio, 128GB. Main model: https://huggingface.co/lmstudio-community/Qwen2.5-Coder-32B-Instruct-MLX-4bit Draft model: https://huggingface.co/lmstudio-community/Qwen2.5-Coder-0.5B-Instruct-MLX-4bit or https://huggingface.co/mlx-community/Qwen2.5-0.5B-Instruct-4bit Prompt: "Write a quicksort algorithm"...