kernel-memory icon indicating copy to clipboard operation
kernel-memory copied to clipboard

Make ITextEmbeddingGenerator.CountTokens and ITextGenerator.CountTokens ValueTask<int>

Open JohnGalt1717 opened this issue 6 months ago • 5 comments

Right now these are synchronous, but if you're using an online service to implement these (i.e. LLama.cpp server) then these need to be able to return async responses. Having it return ValueTask would be greatly helpful.

Inversely GenerateEmbeddingAsync could be ValueTask as well.

JohnGalt1717 avatar Dec 14 '23 14:12 JohnGalt1717