TextChunker CPU usage

Open xkzhud opened this issue 1 year ago • 2 comments

Context / Scenario

finally,i found the problem. when excuting partition.

What happened?

when excuting partition step , the cpu increae from 0.06% to 46.6%. and the speed is slow. this perfomance problem may make km unusable. the cpu cost should be reduce and the time cost should be reduce.

Importance

I cannot use Kernel Memory

Platform, Language, Versions

.net8

Relevant log output

No response

Mar 18 '24 03:03 xkzhud

Could you provide a file to reproduce the problem?

Mar 18 '24 09:03 dluc

sure .here is the doc Uploading 题.docx…

Mar 19 '24 02:03 xkzhud

TODO: work with tokens, chunking/counting only once. To be part of semantic chunker feature.

Jul 09 '24 14:07 dluc