sparseml
sparseml copied to clipboard
Log overall sparsity of the model
Issues like https://github.com/neuralmagic/sparseml/issues/1282 are very hard to detect when global_sparsity: True, as the final model will have the desired target sparsity but the sparsity scheduler might have followed the wrong interpolation function. This PR adds logging of the overall sparsity of the model, which makes this issue easily detectable.
@bfineran that's a very good point for multiple modifiers, I've completely missed it.
Regarding logging only when global_sparsity is True:
- from practical point of view, I think it would be nice to have all runs logging the same things so it is easier to compare them visually in e.g. wandb dashboard
- even if we log only when
global_sparsityis True, multiple modifiers issue can happen because SparseML notion ofglobal_sparsitydoesn't mean global for the entire model (implying there will be only one modifier); it means the sparsity profile is global only for the givenparamsin the specific PruningModifier. There can be multiple global-pruners which are pruning different subsets ofparamswithglobal_sparsity: True. - the latest commit attempts to solve this by logging sparsity with the name of the respective PruningModifier class