SHARK
SHARK copied to clipboard
Add some metadata and features to benchmark results.
- Rename "dynamic" bool field to "shape_type" and provide "static" or "dynamic" as values.
- Provide datatype in benchmark results.
- Computes and provides comparison with PyTorch (or Tensorflow) for shark-python and shark-iree-c benchmarks (e.g. 2x slower/faster)
- Adds tags and notes to tank/pytorch/torch_model_list.csv to be added to the first entry row (PT/TF benchmark) for every benchmarked test case.
Example:
model,engine,dialect,device,shape_type,data_type,iter/sec,ms/iter,vs. PyTorch/TF,iterations,param_count,tags,notes,datetime
microsoft/MiniLM-L12-H384-uncased,tensorflow,mhlo,cpu,static,int32,17.557325179243154,56.956284046173096,=,100,66M,"nlp,bert-variant,transformer-encoder","Large version has 12 layers, 384 hidden size,Smaller than BERTbase (66M params vs 109M params)",2022-08-27 02:41:01.053269
microsoft/MiniLM-L12-H384-uncased,shark_python,mhlo,cpu,static,int32,31.806993922301466,31.439626216888428,1.81x faster,100,,,,2022-08-27 02:41:04.414347
microsoft/MiniLM-L12-H384-uncased,shark_iree_c,mhlo,cpu,static,int32,36.36363636363637,27.499999999999996,2.07x faster,100,,,,2022-08-27 02:41:05.494994