preprocess
preprocess copied to clipboard
Add -c option to split-sentences.perl
Some documents contain extremely long lines of generated text (most often links to search page results) that take forever to parse with the regular expressions in split-sentences.perl. Using the -c option these lines can be completely ignored.
Ideally we'd replace buffering then splitting with splitting on the fly. Then if there's something long and no split we throw it out. Here I'm a bit concerned we're throwing out stuff that would correctly split. I understand your immediate need though.