benchmark
benchmark copied to clipboard
Update score and plot to compute subscore for gpu/cpu/train/eval
The current torchbenchmark framework reports one single benchmark score and doesn’t give a way to calculate the subscore for GPU/CPU/train/eval/eager/JIT. So we need a way to get the benchmark subscore by calculating the subscore for each of the following 8 configurations:
* train-cuda-eager
* train-cpu-eager
* eval-cuda-eager
* eval-cpu-eager
* train-cuda-jit
* train-cpu-jit
* eval-cuda-jit
* eval-cpu-jit
Here are the two subtasks required to accomplish this:
- [ ] Update the subscore depending on the configuration
- [ ] Generate 9 plots . One for each of the above configuration(8) and then one with the whole score