GLiNER icon indicating copy to clipboard operation
GLiNER copied to clipboard

Published 20 hours ago • verified publisher icon urchade

Chunking for the 384 words limit

Open rjalexa opened this issue 9 months ago • 4 comments

What is the best way to chunk longer texts so each chunk fits under the 384 words (or 512 subtokens) ? My articles on the average are around 1200 tokens / 5000 chars approx Thank you

rjalexa avatar May 08 '24 12:05 rjalexa