sparseml icon indicating copy to clipboard operation
sparseml copied to clipboard

Update SparseGPT updates to respect base model's sparsity

Open abhinavnmagic opened this issue 1 year ago • 0 comments

Motivation

If we run SparseGPT on a base model at some sparsity, the sparsity mask after SparseGPT could be very different from the initial one. In other words, SparseGPT doesn't preserve the initial sparsities of the base model. This PR fixes that. If preserve_sparsity_mask=True in the recipe, the sparsity mask of the base model being pruned is preserve. This can be useful for multiple scenarios: one-shot from 50->70% sparsity, going from unstructured to structured sparsity or vice versa, etc.

Testing

The implementation was tested by applying 2:4 sparsity on a 70% sparse model. The resulting model was checked if it preserves the 70% sparsity masks (ie. weights which were pruned out in the base model remain pruned), and if it has 2:4 sparsity pattern.

Recipe

test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: 0.5
      block_size: 128
      sequential_update: false
      quantize: false
      percdamp: 0.01
      mask_structure: "2:4"
      targets: ["re:model.layers.\\d*$"]
      preserve_sparsity_mask: true

abhinavnmagic avatar Apr 04 '24 22:04 abhinavnmagic