Add Model Zoo testing support
This change adds 2 "Model Zoo" to test:
- ONNX Zoo (https://onnx.ai/models/ and https://github.com/onnx/models)
- The script will use the repository, and does automatically the following steps:
- Downloads the models one-by-one with the provided test sample
- Unzips and converts everything to test_runners.py format
- Runs the test
- Cleans up after the run
- Repeats for all selected models
- The script will use the repository, and does automatically the following steps:
- Sample Generator with Datasets
- The provided pyhon modul can download models and generated test samples from real datasets for them
- It is extendable with furter datasets and models
- Current support
- 3 Datasets for image, text and audio
- ImageNet 2012 Val, SQuAD v1.1, LibriSpeech ASR
- Models using those datasets
- ResNet50 v1, ResNet50 v1.5, TIMM MobileNetv3-large, ViT-base-patch16-224, CLIP-ViT-large-patch14
- BERT-large-uncased, DistilBERT-base-cased-distilled, Roberta-base, GPT-J, Llama2-7b-chat-hf, T5-base, Gemma-2b-it
- Wav2Vec2-base-960h, Whisper-small-en
- 3 Datasets for image, text and audio
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 92.10%. Comparing base (
3e65032) to head (77bec3a). Report is 3 commits behind head on develop.
:exclamation: Current head 77bec3a differs from pull request most recent head 41d9678
Please upload reports for the commit 41d9678 to get more accurate results.
Additional details and impacted files
@@ Coverage Diff @@
## develop #2990 +/- ##
===========================================
- Coverage 92.16% 92.10% -0.07%
===========================================
Files 493 493
Lines 19690 19645 -45
===========================================
- Hits 18148 18094 -54
- Misses 1542 1551 +9
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
It is working without any errors. By default, the numpy 2.0 is there but it requires numpy 1.24.1
Oh right, it was released not long ago. Probably everything needs to be pinned to a version, not just the critical part like torch.