lora icon indicating copy to clipboard operation
lora copied to clipboard

Published 20 hours ago • verified publisher icon cloneofsimo

About SVD Distillation

Open zhihongp opened this issue 2 years ago • 1 comments
trafficstars

Great work. A pity that I am late on this so this might be a naïve question but it will help me greatly.

What is the SVD distillation (as mentioned in README) for? As I understand, with LoRA, the dimensionality reduction is already done in the form of low rank matrices A & B. What is SVD (singular value decomposition) distillation applied to?

zhihongp avatar Mar 24 '23 23:03 zhihongp

See this discussion: https://github.com/cloneofsimo/lora/discussions/56#discussioncomment-4510380

cloudhan avatar May 05 '23 08:05 cloudhan