Request: Token Count on Gemini (all) APIs

[joe-sage]

Hola,

First off, very cool library! Thank you for sharing.

I haven't delved too deep into the codebase, and may just fork and add this feature myself if it isn't planned, but I'd like to retrieve token counts from the ai providers (specifically Gemini). I'm unsure how that interacts with the library, though. Does each 'pass' send the entire document, prompt and all, to the API, so each call would be multiplied by the number of passes?

Either way, are token counts planned to be included generally?

Thanks again!

[aksg87]

Thanks! and appreciate the question. Gemini and other models allow you to calculate tokens in some text using their API example. I think it's a nice idea to incorporate this in a general way with each model as part of the response, but this is likely less of a priority than some of the other issues and requests.

Regarding document processing, the library breaks documents down based on chunk size in characters, processes each chunk in parallel, and then merges the results. If you increase the number of passes, each chunk is processed multiple times for improved recall. Hopefully, we can have more in-depth documentation on this soon, with a visual soon.

[bikubay]

even as we wait for this enhancement, is there a rough estimate of how much increase in token we shoudl expect, a simple formula like input token * number of passes + ??? would be very helpful