sparseml icon indicating copy to clipboard operation
sparseml copied to clipboard

[GPT pruning] Add SparseML support for CLM tasks

Open eldarkurtic opened this issue 2 years ago • 0 comments

This PR adds SparseML support for causal language modelling tasks from HuggingFace transformers library. This is used to prune generative language models, e.g. GPT-style models.

Apart from the standard SparseML args and HF defaults, this script adds support to select eval_split which is useful to quickly switch evaluations from validation to test splits.

It has been tested with simple GMP pruner, but also with our upcoming ZipGPT2 pruning approach.

eldarkurtic avatar Jan 31 '23 23:01 eldarkurtic