Benchmarking

[Fei-Guo]

[Fei-Guo]

We need to onboard benchmarks to evaluate the effectiveness/performance of model inference/fine tuning or RAG.

[rudrakshkarpe]

Hi @Fei-Guo,

Do you have any frameworks in mind to evaluate RAG pipelines?

Ragas is a popular one, we can try integrating it, any thoughts on this?

[bangqipropel]

@rudrakshkarpe We will consider to run this, thanks!

[rudrakshkarpe]

My pleasure, I'd love to contribute to onboarding benchmarks and other technical aspects of this project. Do you guys have any community chat channel or community meetings?

[sdesai345]

To-do: extend perf benchmarking to general purpose CPU compute, for users interested in deploying to more cost-effective and widely-available CPU resources (compared to GPU)