Automating benchmark testing with every pr merged.

[Manan17]

Hello, So we discussed an approach to solve this: Running all the benchmarks take less than an hour. I tried it on a single H100 GPU and it took me 55 minutes. So, what we thought that we can run the benchmark scripts after every pr is merged to the main repo using the CI. Similar to what we do for docs, we will run a the benchmark script after anything is pushed to the main branch of the repo. Initially the data of the benchmark will be stored in the gh-pages branch. Once the script is run (make run-benchmarks) the new data will be appended to the csv file in the gh-pages branch with another column which would be github-commit-hash. So now, for every commit we will have the benchmark data, and it will be visualized using github pages on for example: https://linkedin.github.io/Liger-Kernel/benchmarks. Using javascript the data will be visualized using charts just like (https://hud.pytorch.org/benchmark/compilers). There would be filters and user can see benchmark improvements for speed and memory. Let me know what you guys think about this. @Tcc0403 @yundai424 @lancerts