sparseml
sparseml copied to clipboard
Implement mask-transfer between models at init
- masks are transferred from the given model at initialization
- to keep masks on, one should add ConstantPruningModifier in the yaml recipe
- add apply_weight_mask function to transfer masks between models
- add command-line argument --apply_mask_from to all Transformers examples
- useful for: mask-only transfer experiments (for example lottery-ticket style)
@bfineran I have pulled out transfer of masks to the SparseAutoModel class, to avoid repeating a bunch of code for each task. Now, one could enable this feature when adding new tasks by including this snippet:
if mask_name_or_path:
SparseAutoModel._transfer_masks(
mask_name_or_path,
model_name_or_path,
model,
delayed,
)
Logging info is also included in it.
@markurtz makes sense, given that this feature would do the same thing for all models. What do you mean by CLI utility script? The only potential issue I can see with generalizing this feature is model loading, given that they can have unique loading procedures and should probably be implemented on a case-by-case basis.