Bad Bot Blocker

[Ahmadshoh]

Is your feature request related to a problem? Please describe.

The current setup of nginx-proxy-manager lacks built-in support for integrating the nginx-ultimate-bad-bot-blocker module, resulting in a gap in security measures against malicious bots. Without this integration, administrators may face challenges in effectively mitigating bot-related threats and protecting their systems and applications from potential vulnerabilities and performance issues.

Describe the solution you'd like

I propose adding native support for enabling the nginx-ultimate-bad-bot-blocker within nginx-proxy-manager. This would involve integrating the functionality of the bot blocker module directly into nginx-proxy-manager's interface, allowing administrators to easily activate and configure bot blocking settings without the need for manual configuration or separate installations. This enhancement would streamline the process of fortifying web servers against malicious bot activities, enhancing security and performance for users of nginx-proxy-manager.

Describe alternatives you've considered

One alternative approach would be for administrators to manually configure the nginx-ultimate-bad-bot-blocker module alongside nginx-proxy-manager. However, this approach requires additional technical expertise and may introduce complexities in managing the integration between the two components. Additionally, manual configuration increases the risk of misconfigurations and potential conflicts between settings, which could impact system stability and security.

Additional context

Integrating the nginx-ultimate-bad-bot-blocker module into nginx-proxy-manager would provide a comprehensive solution for protecting web servers against a wide range of bot-based threats, including web scraping, DDoS attacks, and vulnerability scanning. By incorporating this functionality directly into nginx-proxy-manager's interface, administrators can easily enable and customize bot blocking settings, enhancing the overall security posture of their infrastructure. Additionally, the integration would align with the goal of nginx-proxy-manager to provide a user-friendly interface for managing NGINX configurations, simplifying the implementation of advanced security measures for both novice and experienced users.

[Ahmadshoh]

More info you can find here:

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker

[github-actions[bot]]

Issue is now considered stale. If you want to keep it open, please comment :+1:

[gizmocuz]

Would still be great if this would be supported

[eikaramba]

yes please, metas new crawler are extremly annoying and do not respect the robots.txt - i am flooded with multiple requests per second and various domains

in the meantime i added this to the custom nginx configuration under advanced:

# Block various bots
if ($http_user_agent ~* (meta-externalagent|facebookexternalhit|facebookbot|facebook|crawler|bot|spider|crawl)) {
    return 403;
}

[ichilver]

This would be useful. I found this https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker but couldn't see how this would work with an existing NPM docker image.

[Kisaragi-ng]

would love to have this feature, has anyone tried to use the manual configuration method?

[ErroneousBosch]

Bumping that there should be a plugin or something for this. Even an ability to put in custom configurations. I threw together a quick and dirty shell script that should work for bots and IP addresses, I may revisit it later this week.

it generates files that can be placed in /data/nginx/custom/ that *should * get loaded in.

#! /bin/bash
# this script generates bad bot blocking configurations for nginx proxy manager
# grab latest version of known bad bots from https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list and map to an array
# Download the bad user agents list
bad_bots_list=$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list)

# Map the list to an array
IFS=$'\n' read -r -d '' -a bad_bots_array <<< "$bad_bots_list"

# Generate the config file
# Create the config file
CONFIGFILE="http_top.conf"
echo "# This file is generated by generate_configs.sh" > $CONFIGFILE
 echo "map \$http_user_agent \$bad_bot {" >> $CONFIGFILE
	echo "  default		0;" >> $CONFIGFILE
# Loop through the array and generate the config
for bot in "${bad_bots_array[@]}"; do
    # Remove any leading or trailing whitespace
    bot=$(echo "$bot" | xargs)
    # Skip empty lines
    if [ -z "$bot" ]; then
        continue
    fi
    # Generate the config for the bot
    echo "  \"~*(?:\b)$bot(?:\b)\"		3;" >> $CONFIGFILE
done
# Close the map block
{
  echo "}" 
  echo "geo \$validate_client {" 
  echo "  default		0;" 
}>> $CONFIGFILE
bad_ip_list=$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-ip-addresses.list)
bad_ip_list+=$'\n'$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/fake-googlebots.list)
bad_ip_list+=$'\n'$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/nibbler-seo.list)
bad_ip_list+=$'\n'$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/seo-analysis-tools.list)

# Map the list to an array
IFS=$'\n' read -r -d '' -a bad_ip_array <<< "$bad_ip_list"
# Loop through the array and generate the config
for ip in "${bad_ip_array[@]}"; do
    # Remove any leading or trailing whitespace
    ip=$(echo "$ip" | xargs)
    # Skip empty lines
    if [ -z "$ip" ]; then
        continue
    fi
    # Generate the config for the bot
    echo "  $ip		1;" >> $CONFIGFILE
done
good_ip_list=$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/google-ip-ranges.list)
good_ip_list+=$'\n'$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/cloudflare-ip-ranges.list)
good_ip_list+=$'\n'$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bunnycdn-net.list)
good_ip_list+=$'\n'$(curl -s https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bing-ip-ranges.list)

# Map the list to an array
IFS=$'\n' read -r -d '' -a good_ip_array <<< "$good_ip_list"
# Loop through the array and generate the config
for ip in "${good_ip_array[@]}"; do
    # Remove any leading or trailing whitespace
    ip=$(echo "$ip" | xargs)
    # Skip empty lines
    if [ -z "$ip" ]; then
        continue
    fi
    # Generate the config for the bot
    echo "$ip		0;" >> $CONFIGFILE
done
# Close the geo block
echo "}" >> $CONFIGFILE

CONFIGFILE="server_proxy.conf"
echo "# This file is generated by generate_configs.sh" > $CONFIGFILE
{
  echo "if (\$bad_bot = '3') {" 
  echo "  return 444;" 
  echo "}" 
  echo "if (\$validate_client) {"
  echo "  return 444; "
  echo "}"

} >> $CONFIGFILE

[github-actions[bot]]

Issue is now considered stale. If you want to keep it open, please comment :+1:

[gizmocuz]

I know you're very busy converting the project to React... But just commenting to keep this open...