sparseml
sparseml copied to clipboard
[GPT pruning] Add SparseML support for CLM tasks
This PR adds SparseML support for causal language modelling tasks from HuggingFace transformers library. This is used to prune generative language models, e.g. GPT-style models.
Apart from the standard SparseML args and HF defaults, this script adds support to select eval_split which is useful to quickly switch evaluations from validation to test splits.
It has been tested with simple GMP pruner, but also with our upcoming ZipGPT2 pruning approach.