OpenAI Tokens Usage for RAGAs Evaluation

[aliai22]

I am using RAGAs evaluation metrics to evaluate my RAG pipeline. Is there any way I could know the openai tokens used per api call from RAGAs, lets say for any one metric, i.e., faithfulness. I want to estimate the cost of using RAGAs evaluators.

[jjmachan]

this is a valuable suggestion @aliai22 :)

we have thought about it and will be implementing it soon but it might take a couple of weeks to prioritize this. The reason is that I want to take some time to design how stats would be like

basically, we want to track things like

• tokens used
• cost
• performance metrics etc

in a unified way. I'll explore more and send an RFC your way asking for your feedback as soon as I have it ready. would that help?

[aliai22]

Yeah, that would be great. Looking forward to hear from you.

[Vishnu-add]

Did it get implemented?

[fhuthmacher]

I am also looking for this, any update?

[abetatos]

Pinging to see if this gets released at any point.

[nfsergiu]

Any updates on this @jjmachan ?

[whitney-meer]

I'm also interested in this!

[jjmachan]

hey folks, not yet but I will take this up this week and have something ready for your feedback shortly, my sincere apologies for the delay!

[jjmachan]

fixed in #1129