sparseml icon indicating copy to clipboard operation
sparseml copied to clipboard

Implement mask-transfer between models at init

Open eldarkurtic opened this issue 3 years ago • 2 comments

  • masks are transferred from the given model at initialization
  • to keep masks on, one should add ConstantPruningModifier in the yaml recipe
  • add apply_weight_mask function to transfer masks between models
  • add command-line argument --apply_mask_from to all Transformers examples
  • useful for: mask-only transfer experiments (for example lottery-ticket style)

eldarkurtic avatar Apr 22 '22 01:04 eldarkurtic

@bfineran I have pulled out transfer of masks to the SparseAutoModel class, to avoid repeating a bunch of code for each task. Now, one could enable this feature when adding new tasks by including this snippet:

        if mask_name_or_path:
            SparseAutoModel._transfer_masks(
                mask_name_or_path,
                model_name_or_path,
                model,
                delayed,
            )

Logging info is also included in it.

eldarkurtic avatar May 06 '22 09:05 eldarkurtic

@markurtz makes sense, given that this feature would do the same thing for all models. What do you mean by CLI utility script? The only potential issue I can see with generalizing this feature is model loading, given that they can have unique loading procedures and should probably be implemented on a case-by-case basis.

eldarkurtic avatar May 26 '22 16:05 eldarkurtic