Robots.txt-Parser-Class
Robots.txt-Parser-Class copied to clipboard
Adds a filter that guarantees to return batches ending with a line br…
In case of very large robots.txt files (https://edu-medic.ru/robots.txt) standard batch size (4096 bytes) can lead to incorrect tree building. Because line can be divided at random position.
At the very beginning of filters chain new filter was added. It ensures, that all next filters will receive lines ending with EOL symbol.