LLMUnity
LLMUnity copied to clipboard
How to Calculate Token Score (Token / minute) to check if the CPU is strong enough?
Describe the feature
Hi, your unity bindings for Llama.cpp is great. I'm wondering is there any equivilant method in this SDK which I can use to calculate token score? This is how this android based library doing this.
suspend fun myCustomBenchmark(): Flow<String> = flow {
try {
withTimeout(30.seconds) { // Set timeout to 2 minutes
when (val state = threadLocalState.get()) {
is State.Loaded -> {
val ncur = IntVar(completion_init(state.context, state.batch, "Write an article on global warming in 1000 words", nlen))
while (ncur.value <= nlen) {
val str = completion_loop(state.context, state.batch, state.sampler, nlen, ncur)
if (str == null) {
_isSending.value = false
_isCompleteEOT.value = true
break
}
if (stopGeneration) {
break
}
emit(str)
}
kv_cache_clear(state.context)
}
else -> {
_isSending.value = false
}
}
}
} catch (e: Exception) {
// Handle timeout or any other exceptions if necessary
if (e is kotlinx.coroutines.TimeoutCancellationException) {
println("Benchmark timed out after 2 minutes.")
}
} finally {
_isSending.value = false
}
}.flowOn(runLoop)
Does your library also has the equivilant method to check benchmarks (token per minute)? Thanks.