[BUG] bot2_reqlimitip Banning IP 127.0.0.1

[tristanbes]

Hello,

On a Debian 10, running Varnish -> Nginx -> php-fpm

The nginx error logs reports:

2019/07/25 15:22:48 [error] 10057#10057: *403175 limiting requests, excess: 10.024 by zone "bot2_reqlimitip", client: 127.0.0.1, server: , request: "GET /media/original/question-138985.jpg HTTP/1.1", host: "actus-cabinet.fr"

In /etc/nginx/bots.d/whitelist-ips.conf I have:

# Add One Entry Per Line - Can Include Ranges like 127.0.0.1/32
127.0.0.1 0;

1. Why is the varnish IP is picked instead of the real user IP ?
2. Why is the IP whitelist being ignored ?

PS: I use this project since few years, so maybe it's related to an update or a change somewhere ?

thank you :)

[mitchellkrogza]

Hi @tristanbes sorry you are having trouble. I did do extensive testing on the whitelisting system and introduced tests which fail the build if the whitelist tests fail.

First can you configure nginx so that the real IP is not hidden. Here's a guide to do that

https://ypereirareis.github.io/blog/2017/02/15/nginx-real-ip-behind-nginx-reverse-proxy/

Once you have that detecting which IP is being rate restricted may be easier.

I did make changes to the bot2_reqlimitip zone which I will review and revert back to in the morning.

Yannick PEREIRA-REIS

Get user real ip in nginx behind nginx reverse proxy

Behind a reverse proxy, the user IP we get is often the reverse proxy IP itself. But for obvious reasons it's important to have access to the user real ip address.

[tristanbes]

Weird, I had on the vhost

set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;

I'll continue to monitor the situation; Maybe try to disable rate throttling to see if it's what's causing our prod downtime/slow response time.

Thank you

[mitchellkrogza]

Just increase, for now, manually, the zone info for bot2_reqlimitip zone settings at the bottom of globalblacklist.conf change it from 30r/m to something like 30r/s until I can review it in the morning. Just remember when you update it will wipe out any mods to that. But don't stress I will have it sorted in the morning

[tristanbes]

ok thanks, same, don't stress, i'm not sure what's happening is really caused by your code.

[itoffshore]

@tristanbes - I think 127.0.0.1 should be whitelisted - the slow response is because nginx is rate limiting the connections from the reverse proxy.

Rate limiting should also be done in iptables

[mitchellkrogza]

@tristanbes please update to latest version and let me know if this has addressed the problem.

[tristanbes]

@itoffshore 127.0.0.1 was already whitelisted as i stated on my original message (see: whitelist-ips.conf)

Will plan an upgrade early next week and see if I can see 2019/07/25 15:22:48 [error] 10057#10057: *403175 limiting requests, excess: 10.024 by zone "bot2_reqlimitip", client: 127.0.0.1, server:... message again.

Thank you

[tristanbes]

@mitchellkrogza After checking, when I noticed the message I posted regarding 127.0.0.1 being limited because of the zone "bot2_reqlimitip" I was still on version 2 of your library, :).

[mitchellkrogza]

@tristanbes grab the latest globalblacklist.conf and all the required includes.

Everything has been updated extensively so its probably best to do a full update of everything but just back up your existing stuff so you can once again customize your new includes from the old ones.

There's been substantial improvements since V2, whitelisting especially was NOT working properly and is now 100% working and thoroughly tested too. A number of bugs were addressed and every single aspect of the blocker thoroughly tested on each and every build.

[tristanbes]

Good to know the upgraded is needed :) And it's still possible to use only what we want ? By memory I used only the badbots list and the request limits;