Robots.txt-Parser-Class icon indicating copy to clipboard operation
Robots.txt-Parser-Class copied to clipboard

Adds a filter that guarantees to return batches ending with a line br…

Open edim24 opened this issue 1 year ago • 0 comments

In case of very large robots.txt files (https://edu-medic.ru/robots.txt) standard batch size (4096 bytes) can lead to incorrect tree building. Because line can be divided at random position.

At the very beginning of filters chain new filter was added. It ensures, that all next filters will receive lines ending with EOL symbol.

edim24 avatar Dec 23 '24 18:12 edim24