deepsparse icon indicating copy to clipboard operation
deepsparse copied to clipboard

[Move] Common Evaluation Modules to `Sparsezoo``

Open rahul-tuli opened this issue 1 year ago • 1 comments

Evaluator Move

This PR moves a few modules from deepsparse.evaluation to sparsezoo.evaluation

Motivation and Context

The moved modules provide a common interface for evaluating models. This interface can be leveraged and used by both SparseML and DeepSparse to evaluate models.

Since SparseZoo is a common dependency for both repositories, it makes sense to move these modules to SparseZoo.

How Has This Been Tested?

The tests for moved modules in deepsparse.evaluation are moved to sparsezoo.evaluation and other deepsparse specific deepsparse.evaluation tests were updated to use the moved modules.

Both deepsparse and sparsezoo tests were run to ensure that the move did not break anything.

Also ran the following eval command:

$ deepsparse.eval --target zoo:mpt-7b-mpt_pretrain-base_quantized \                                                             (move-evaluator-to-sparsezoo|●1✚2…4)
                --dataset hellaswag \
                --dataset gsm8k \
                --integration lm-evaluation-harness \
                --limit 2

Note: This diff relies on sparsezoo diff https://github.com/neuralmagic/sparsezoo/pull/411

The failing tests should pass after sparsezoo side changes are merged!

rahul-tuli avatar Jan 09 '24 15:01 rahul-tuli

base tests failing - LGTM otherwise

bfineran avatar Feb 02 '24 20:02 bfineran