luminal icon indicating copy to clipboard operation
luminal copied to clipboard

Published 20 hours ago • verified publisher icon jafioti

Quantization

Open jafioti opened this issue 1 year ago • 1 comments

  • [x] 8 bit
  • [ ] 4 bit
  • [ ] 2 bit? (Check if accuracy falls)

jafioti avatar Feb 05 '24 15:02 jafioti

Continuing down the quantization scale will require perplexity benchmarking. I'm skeptical of quants below 8 bit for now.

jafioti avatar Feb 27 '24 05:02 jafioti