candle icon indicating copy to clipboard operation
candle copied to clipboard

Implement the Gemma 3 vision models!

Open EricLBuehler opened this issue 11 months ago • 1 comments

  • [x] Add vision part - mmproj
  • [x] Implement forward pass
  • [x] Add inputs processor (pan & scan, normalize, etc)
  • [x] Example

EricLBuehler avatar Mar 17 '25 03:03 EricLBuehler

when i run the 4b and 12b like

cargo run --example gemma_vision --release --features cuda,cudnn -- --prompt "describe this image" --which v3_4b --image /home/pro/Desktop/sd_final.png 

it downloads the models and then i get error

avx: true, neon: false, simd128: false, f16c: true
temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64
retrieved the files in 98.089819ms
Error: data did not match any variant of untagged enum Gemma3Config

AlpineVibrations avatar Mar 18 '25 00:03 AlpineVibrations

is this working? its been sitting here for a while would be great to have Gemma vision models in candle especially if the work is done..?

AlpineVibrations avatar Jun 13 '25 22:06 AlpineVibrations