Jan-Petter Gundersen
Jan-Petter Gundersen
It's PHP 5.4 compatible too, pushed https://github.com/t1gor/Robots.txt-Parser-Class/commit/b74ad596468f4927e0de290f371eb5c27f0b9efb a few minutes later, with an build fix... Still, there is work to do: - [x] Render the Host directive - [ ]...
@szepeviktor I know, but that doesn't fix the inline directives. This is what it outputs: ``` Disallow: clean-param:token path // invalid path Disallow: host:a.b // invalid host ``` It should...
Rendering of the (regular) host directive now implemented. https://github.com/t1gor/Robots.txt-Parser-Class/commit/627082db6d76b1fd31890012d6a65d0e51369642 `Host: example.com` --- I'm not sure where I saw it, but there was some robots.txt files where it was used as...
There's a link on the front page, at the top of the [README.md](https://github.com/t1gor/Robots.txt-Parser-Class/blob/master/README.md) to be exact, the red box, from there it should be pretty easy to get an overview,...
Yeah, why fix this library at all, when [VIPnytt/RobotsTxtParser](https://github.com/VIPnytt/RobotsTxtParser) has everything this library is missing?? I needed to handle caching, proper rendering, automatic download, proper user-agent matching, and it had...
@t1gor You're using the MIT license, witch allows me to do anything really. I've moved the idea, but 95% of the code is unique. The only reasons behind this decision,...
To get completely rid of the performance issues, both this issue and #75 needs to be fixed. I'd also discover this interesting `robots.txt` file: [http://www.goldmansachs.com/robots.txt](http://www.goldmansachs.com/robots.txt) At time of writing, it's...
I've transitioned to an competing library, witch is relatively deeply integrated into an web crawler, running in a production environment. Would love to benchmark, and compare to see if there's...
Give me a week or two, and I'll implement this in production. Sorry for dragging this out in time...
Doing some quick testing, and honestly, I still don't know, if I **ever** can trust this library, in production... Found an pretty big, fundamental and obvious bug. #93 Literally in...