sparseml icon indicating copy to clipboard operation
sparseml copied to clipboard

Implement multiple-choice pipeline

Open eldarkurtic opened this issue 2 years ago • 0 comments

This PR implements multiple-choice pipeline in HF's transformers library with SparseML integration. It enables commonsense-reasoning experiments proposed in the "Sparsity May Cry benchmark".

Supported datasets/tasks

  1. SWAG: https://arxiv.org/abs/1808.05326
  2. Commonsense_QA: https://arxiv.org/abs/1811.00937
  3. RACE: https://arxiv.org/abs/1704.04683
  4. Winogrande: https://arxiv.org/abs/1907.10641

Data preprocessing pipeline for each dataset follows the proposed format from the corresponding paper.

Verification The implementation is tested by reproducing results from the RoBERTa paper (https://arxiv.org/abs/1907.11692) and fairseq repository (https://github.com/facebookresearch/fairseq/tree/main/examples/roberta).

eldarkurtic avatar Dec 27 '22 10:12 eldarkurtic