sparseml icon indicating copy to clipboard operation
sparseml copied to clipboard

Log overall sparsity of the model

Open eldarkurtic opened this issue 2 years ago • 1 comments

Issues like https://github.com/neuralmagic/sparseml/issues/1282 are very hard to detect when global_sparsity: True, as the final model will have the desired target sparsity but the sparsity scheduler might have followed the wrong interpolation function. This PR adds logging of the overall sparsity of the model, which makes this issue easily detectable.

eldarkurtic avatar Dec 22 '22 09:12 eldarkurtic

@bfineran that's a very good point for multiple modifiers, I've completely missed it.

Regarding logging only when global_sparsity is True:

  • from practical point of view, I think it would be nice to have all runs logging the same things so it is easier to compare them visually in e.g. wandb dashboard
  • even if we log only when global_sparsity is True, multiple modifiers issue can happen because SparseML notion of global_sparsity doesn't mean global for the entire model (implying there will be only one modifier); it means the sparsity profile is global only for the given params in the specific PruningModifier. There can be multiple global-pruners which are pruning different subsets of params with global_sparsity: True.
  • the latest commit attempts to solve this by logging sparsity with the name of the respective PruningModifier class

eldarkurtic avatar Dec 26 '22 06:12 eldarkurtic