AMDMIGraphX icon indicating copy to clipboard operation
AMDMIGraphX copied to clipboard

Add Model Zoo testing support

Open attila-dusnoki-htec opened this issue 1 year ago • 1 comments

This change adds 2 "Model Zoo" to test:

  • ONNX Zoo (https://onnx.ai/models/ and https://github.com/onnx/models)
    • The script will use the repository, and does automatically the following steps:
      • Downloads the models one-by-one with the provided test sample
      • Unzips and converts everything to test_runners.py format
      • Runs the test
      • Cleans up after the run
      • Repeats for all selected models
  • Sample Generator with Datasets
    • The provided pyhon modul can download models and generated test samples from real datasets for them
    • It is extendable with furter datasets and models
    • Current support
      • 3 Datasets for image, text and audio
        • ImageNet 2012 Val, SQuAD v1.1, LibriSpeech ASR
      • Models using those datasets
        • ResNet50 v1, ResNet50 v1.5, TIMM MobileNetv3-large, ViT-base-patch16-224, CLIP-ViT-large-patch14
        • BERT-large-uncased, DistilBERT-base-cased-distilled, Roberta-base, GPT-J, Llama2-7b-chat-hf, T5-base, Gemma-2b-it
        • Wav2Vec2-base-960h, Whisper-small-en

attila-dusnoki-htec avatar Apr 23 '24 10:04 attila-dusnoki-htec

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.10%. Comparing base (3e65032) to head (77bec3a). Report is 3 commits behind head on develop.

:exclamation: Current head 77bec3a differs from pull request most recent head 41d9678

Please upload reports for the commit 41d9678 to get more accurate results.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2990      +/-   ##
===========================================
- Coverage    92.16%   92.10%   -0.07%     
===========================================
  Files          493      493              
  Lines        19690    19645      -45     
===========================================
- Hits         18148    18094      -54     
- Misses        1542     1551       +9     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Apr 23 '24 11:04 codecov[bot]

It is working without any errors. By default, the numpy 2.0 is there but it requires numpy 1.24.1

Oh right, it was released not long ago. Probably everything needs to be pinned to a version, not just the critical part like torch.

attila-dusnoki-htec avatar Jun 26 '24 18:06 attila-dusnoki-htec