uniflow-llm-based-pdf-extraction-text-cleaning-data-clustering
uniflow-llm-based-pdf-extraction-text-cleaning-data-clustering copied to clipboard
Bug: RecursiveSplitter removes all spaces
🐛 Describe the bug
If I manually set the chunk_size in RecursiveSplitter to 50, it would remove all blank spaces.
Running this input:
One of the most important things I didn't understand about the world when I was a child is the degree to which the returns for performance are superlinear.
Teachers and coaches implicitly told us the returns were linear. "You get out," I heard a thousand times, "what you put in." They meant well, but this is rarely true. If your product is only half as good as your competitor's, you don't get half as many customers. You get no customers, and you go out of business.
It's obviously true that the returns for performance are superlinear in business. Some think this is a flaw of capitalism, and that if we changed the rules it would stop being true. But superlinear returns for performance are a feature of the world, not an artifact of rules we've invented. We see the same pattern in fame, power, military victories, knowledge, and even benefit to humanity. In all of these, the rich get richer.
You can't understand the world without understanding the concept of superlinear returns. And if you're ambitious you definitely should, because this will be the wave you surf on.
One of the most important things I didn't understand about the world when I was a child is the degree to which the returns for performance are superlinear.
Teachers and coaches implicitly told us the returns were linear. "You get out," I heard a thousand times, "what you put in." They meant well, but this is rarely true. If your product is only half as good as your competitor's, you don't get half as many customers. You get no customers, and you go out of business.
I got this output:
[[{'output': [{'text': ["OneofthemostimportantthingsIdidn'tunderstandabout", 'theworldwhenIwasachildisthedegreetowhichthereturns', 'forperformancearesuperlinear.', 'Teachersandcoachesimplicitlytoldusthereturnswere', 'linear', '"Yougetout,"Iheardathousandtimes,"whatyouputin."', 'Theymeantwell,butthisisrarelytrue', "Ifyourproductisonlyhalfasgoodasyourcompetitor's,", "youdon'tgethalfasmanycustomers", 'You get no customers, and you go out of business.', "It'sobviouslytruethatthereturnsforperformanceare", 'superlinearinbusiness', 'Somethinkthisisaflawofcapitalism,andthatifwe', 'changedtherulesitwouldstopbeingtrue', 'Butsuperlinearreturnsforperformanceareafeatureof', "theworld,notanartifactofruleswe'veinvented", 'Weseethesamepatterninfame,power,militaryvictories,', 'knowledge,andevenbenefittohumanity', 'In all of these, the rich get richer.', "Youcan'tunderstandtheworldwithoutunderstandingthe", 'conceptofsuperlinearreturns', "Andifyou'reambitiousyoudefinitelyshould,because", 'thiswillbethewaveyousurfon.', "OneofthemostimportantthingsIdidn'tunderstandabout", 'theworldwhenIwasachildisthedegreetowhichthereturns', 'forperformancearesuperlinear.', 'Teachersandcoachesimplicitlytoldusthereturnswere', 'linear', '"Yougetout,"Iheardathousandtimes,"whatyouputin."', 'Theymeantwell,butthisisrarelytrue', "Ifyourproductisonlyhalfasgoodasyourcompetitor's,", "youdon'tgethalfasmanycustomers", 'You get no customers, and you go out of business.']}]}]]
Versions
0.0.30
Code PR is needed