torchtune icon indicating copy to clipboard operation
torchtune copied to clipboard
verified publisher icon pytorch

Benchmark QPS

Open rohan-varma opened this issue 2 years ago • 2 comments

  • We should benchmark our QPS against other libraries such as HF, lit-gpt to see if we are in the same ballpark and understand our perf competitiveness.

rohan-varma avatar Jan 03 '24 08:01 rohan-varma

Great to benchmark, but I feel obligated to say that simplicity and memory-efficiency are are top priority > perf.

joecummings avatar Jan 03 '24 15:01 joecummings

@joecummings agreed, but with the caveat that we can't have sizable gaps on perf wrt competition. Benchmarking this is a milestone for the MVP.

kartikayk avatar Feb 25 '24 20:02 kartikayk