robotstxt

robotstxt copied to clipboard

The parser can panic if it tries to slice a string within char boundaries. ``` thread '' panicked at 'byte index 58198 is not a char boundary; it is inside...

samuelfekete

Document test dependencies

3

While following README.md and trying to build && run official google tests I get this error `/usr/bin/ld: cannot find -labsl::container collect2: error: ld returned 1 exit status` on `make` stage...

let4be

Why does allowed_by_robots and one_agent_allowed_by_robots parse robotstxt for each request?

2

This API and example are really confusing... Why cannot we simply parse once and then call methods to check if URL is allowed?...

let4be