refiners

refiners copied to clipboard

Context

Still working on #165

Want to explore the lora/non-lora in the cross-attention adapter and to replace, conditionnaly

fl.Linear(
  ...
  bias=text_cross_attention.use_bias
)

with

fl.LinearLora(
  ...
  bias=text_cross_attention.use_bias
)

Remarks

• impacted also Conv2DLora for coherence
• Default values are False/False for Lora and True/True for non-Lora layers, i feel it makes sense considering bias on LinearLora is not the rule
• bias only impact the second lora layer, first lora layer bias are non-relevant on my POV