Sachin Raja comments

Results 6 comments of


                                            Sachin Raja

Does MLX-VLM support fine-tuning of vision tower layers?

This is very helpful. Many thanks @Blaizzy ! Eagerly looking forward to the overhauled trainer.

Does MLX-VLM support fine-tuning of vision tower layers?

@Goekdeniz-Guelmez Thank you so much for your contribution. Quick question: For optimizing memory utilization, would quantized full weights fine tuning be supported?

Error in FineTuning deepseek-vl-7b-chat-8bit

Thank you!

[BUG] ValueError: [scatter] Cannot calculate VJP with respect to indices.

> It's probably a good idea to add an all zeros vjp w.r.t. indices for scatter like we did for gather for consistency. > > > > Is this for...

[BUG] ValueError: [scatter] Cannot calculate VJP with respect to indices.

Thanks @barronalex , understood. Refering from [this jax implementation](https://github.com/google-research/scenic/blob/main/scenic/model_lib/matchers/hungarian_jax.py#L100), there is another challenge in writing an MLX implementation in python that can be compiled: https://github.com/ml-explore/mlx/issues/1441

[BUG] ValueError: [scatter] Cannot calculate VJP with respect to indices.

Thanks for your response @barronalex ! I realise that HungarianMatcher is not a big performance bottleneck. However, I'm facing memory inflation while in the MLX port of prepare_for_cdn function. https://github.com/ml-explore/mlx/issues/1432...