sparseml
sparseml copied to clipboard
Update SparseGPT updates to respect base model's sparsity
Motivation
If we run SparseGPT on a base model at some sparsity, the sparsity mask after SparseGPT could be very different from the initial one. In other words, SparseGPT doesn't preserve the initial sparsities of the base model. This PR fixes that.
If preserve_sparsity_mask=True in the recipe, the sparsity mask of the base model being pruned is preserve. This can be useful for multiple scenarios: one-shot from 50->70% sparsity, going from unstructured to structured sparsity or vice versa, etc.
Testing
The implementation was tested by applying 2:4 sparsity on a 70% sparse model. The resulting model was checked if it preserves the 70% sparsity masks (ie. weights which were pruned out in the base model remain pruned), and if it has 2:4 sparsity pattern.
Recipe
test_stage:
obcq_modifiers:
SparseGPTModifier:
sparsity: 0.5
block_size: 128
sequential_update: false
quantize: false
percdamp: 0.01
mask_structure: "2:4"
targets: ["re:model.layers.\\d*$"]
preserve_sparsity_mask: true