Jan-Petter Gundersen

I see, but why is the HTML document unencoded before tags are stripped? The contents of the `` tag clearly says `<250A`, so this shouldn't be a problem in the...

@meetmangukiya Thanks, but that didn't really change anything. Beyondpad is definitely more accessible, but the UI is still the same... I've tested with some meta tags, I'm able to set...

Hi @spekulatius You can always enable it by setting `['strict'=>false]` as the 2nd parameter of the constructor, as shown in the [example](https://github.com/VIPnytt/SitemapParser#parsing-of-line-separated-text-strings). Text files has been disabled by default, to...

- GZip files are supported. ref. [/src/SitemapParser.php#L191](https://github.com/VIPnytt/SitemapParser/blob/master/src/SitemapParser.php#L191) - `parseString` is the place to start digging into the code. [/src/SitemapParser.php#L341](https://github.com/VIPnytt/SitemapParser/blob/master/src/SitemapParser.php#L341). I must admit, the current code isn't designed to be this...

Sorry about the missing spec source, here it is: https://developers.google.com/search/reference/robots_txt#file-format > Last updated April 18, 2017.

At the moment, it's possible to generate large (fake or valid) robots.txt files, with the aim to trap the robots.txt crawler, slow down the server, and even cause it to...

I'm quite busy right now, but give me a day or two and I'll come back to this. Stay tuned.

@fte378 Just want to hear your opinion, why does the user-agents _really_ have to be sorted Z-A (witch normally resulting in \* as the last one)? I know there is...

@szepeviktor sorry for the delay, I'll take a look at it

How about now? `$parser->render();`