How to be shure blocker is working ?

[Jonras78]

Hi,

Just got all things set and got the following behavior:

If i do Curl command test by User Agent and Referrer - every time i am getting 200 OK - on bad user agents and referrers. If i adding to Chrome "Custom UserAgent String" extension for testing - blocker blocking bad user agents.

Update: Server logs - with Curl:

HEAD / HTTP/1.1" 200 - "http://100dollars-seo.com" "curl/7.29.0"
HEAD / HTTP/1.1" 200 - "-" "AhrefsBot"

Server logs - with Chrome "Custom UserAgent String" extension:

GET / HTTP/2.0" 403 199 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)"
GET /favicon.ico HTTP/2.0" 404 196 "https://www.domain.com/" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)"

"

Update2: How in logs looks real Ahrefs bot (it should be blocked):

GET /model/chaturbate/gioricci/ HTTP/1.1" 200 194722 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
GET /model/chaturbate/mucmilf59/ HTTP/1.1" 200 194537 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
GET /model/chaturbate/nina_sky_/ HTTP/1.1" 200 194032 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"

What i am missing ? Thanks.

[neerro]

I am interested the same question as I met same problem and misunderstanding.

[mitchellkrogza]

What versions of Apache?

[Jonras78]

Apache version: Apache/2.4.46

[mitchellkrogza]

Apache version: Apache/2.4.46

the only thing that can be wrong is that your <RequireAll> and <RequireAny> structure has been broken higher up or lower down making the blocker (and other security rules you may have) ineffective ... this could be happening in your apache2.conf httpd.conf or in your actual vhost config .... it's so easy to break the 2.4 structure

[neerro]

Apache version: Apache/2.4.46

the only thing that can be wrong is that your <RequireAll> and <RequireAny> structure has been broken higher up or lower down making the blocker (and other security rules you may have) ineffective ... this could be happening in your apache2.conf httpd.conf or in your actual vhost config .... it's so easy to break the 2.4 structure

I believe there are maybe another issues as fact. As I've checked my apache2.conf file many times for both 'RequireAll' and 'RequireAny' structure as well, but it still doesn't work. I've begun using .htaccess file just filled out all your data manually taken from another files and with saving structure inside also excluded all folders and files as you mentioned in yours manuals. And omg it's work fine.

[mitchellkrogza]

Apache version: Apache/2.4.46

the only thing that can be wrong is that your <RequireAll> and <RequireAny> structure has been broken higher up or lower down making the blocker (and other security rules you may have) ineffective ... this could be happening in your apache2.conf httpd.conf or in your actual vhost config .... it's so easy to break the 2.4 structure

I believe there are maybe another issues as fact. As I've checked my apache2.conf file many times for both 'RequireAll' and 'RequireAny' structure as well, but it still doesn't work. I've begun using .htaccess file just filled out all your data manually taken from another files and with saving structure inside also excluded all folders and files as you mentioned in yours manuals. And omg it's work fine.

Getting your RequireAny blocks correct is a nightmare, just one wrong thing will break the entire structure. Every time I run a build here is it tested on both Apache 2.2 and 2.4 before it passes.

[mitchellkrogza]

You can see a log file from the last build here which tests on 2.2 and 2.4 including testing different aspects of the blocker https://travis-ci.org/github/mitchellkrogza/apache-ultimate-bad-bot-blocker/builds/774513497

Travis CI - Test and Deploy Your Code with Confidence

Travis CI enables your team to test and ship your apps with confidence. Easily sync your projects with Travis CI and you'll be testing your code in minutes.

[skonesam]

I'm struggling with the same thing. sudo apache2ctl configtest reports everything is good, but the next tests from a different machine ( like curl -A "80legs" https://usnamemorialhall.org ) happily returns the entire page.

I've added a few IPs to the blocklist as well (3.224.220.101 being one), reloaded apache, and within seconds it's back in the logs.

Running Mediawiki, and the bots are hammering nonsense pages that have the SQL process maxed out as it retrieves diff histories for pages that haven't been edited in years.

Any suggestions? Thanks in advance!