Start benchmarking

[mert-kurttutan]

[mert-kurttutan]

Started this to gather some feedback: Metrics to gather from benchmarking:

1. Speed

2. Accuracy

3. Speed: Two types of measures

• End to end: It is fine to use time function as long as we are measuring process that take at least around 1 sec (otherwise it might become too unreliable to read).
• Finer Measure: This can be done with ``cProfile` package from python that give stats details at the level function calls Speed is also machine dependent (Cpu/Gpu specs, memory). So, it needs these specs In speed measurement, we can also measure speed as we scale different parameters (num_sample, num_features, num_classes)

2. Accuracy: Need to establish some datasets to measure these on.

Another metric: Memory requirements ( but that seems somewhat managed by pretraint constraints)

[noahho]

Dear @mert-kurttutan apologies we didn't have the capacity to work with you on the PR. Thank you for the thoughtful work back then 🙏 We will keep the PR in mind when looking more into the workflow.