crowdsec icon indicating copy to clipboard operation
crowdsec copied to clipboard

Published 20 hours ago • verified publisher icon crowdsecurity

One particular IP ban decision not being banned by bouncer

Open lingfish opened this issue 2 years ago • 14 comments

What happened?

I run a multi server install, and there is one particular IP that keeps getting a ban decision made, but the bouncer (nftables) doesn't ban it because it would seem the IP doesn't show up in /v1/decisions/stream?origins=cscli,crowdsec&startup=true

What did you expect to happen?

The IP to be banned.

How can we reproduce it (as minimally and precisely as possible)?

I'm not entirely sure. Even if I delete the decision manually, it seems to initially block it, and then keep alerting, but not block it.

Anything else we need to know?

The IP is 193.56.29.178. It appears in decision lists, and alerts, but when the bouncer calls the API using /v1/decisions/stream?origins=cscli,crowdsec&startup=true (or false), the IP doesn't appear; other banned IPs do.

The IP does appear in the sqlite DB.

Example, after running a curl against /v1/decisions/stream?origins=cscli,crowdsec&startup=true:

    "new": [
        {
            "duration": "38m55.633568505s",
            "id": 3785624,
            "origin": "crowdsec",
            "scenario": "crowdsecurity/http-probing",
            "scope": "Ip",
            "type": "ban",
            "value": "85.208.136.70"
        },
        {
            "duration": "49m27.565247492s",
            "id": 3785626,
            "origin": "crowdsec",
            "scenario": "crowdsecurity/postscreen-rbl",
            "scope": "Ip",
            "type": "ban",
            "value": "194.87.200.151"
        },
        {
            "duration": "1h27m4.187616928s",
            "id": 3800595,
            "origin": "crowdsec",
            "scenario": "crowdsecurity/postscreen-rbl",
            "scope": "Ip",
            "type": "ban",
            "value": "2.57.122.215"
        },
        {
            "duration": "3h23m56.538359076s",
            "id": 3815577,
            "origin": "crowdsec",
            "scenario": "crowdsecurity/postscreen-rbl",
            "scope": "Ip",
            "type": "ban",
            "value": "103.147.184.194"
        },
        {
            "duration": "3h57m1.756963078s",
            "id": 3815581,
            "origin": "crowdsec",
            "scenario": "crowdsecurity/http-bad-user-agent",
            "scope": "Ip",
            "type": "ban",
            "value": "167.248.133.63"
        }
    ]

Crowdsec version

2023/01/23 16:00:22 version: v1.4.5-debian-pragmatic-a9a2186a76af63551352aa3bc296bdbe80ca4893
2023/01/23 16:00:22 Codename: alphaga
2023/01/23 16:00:22 BuildDate: 2023-01-19_15:06:03
2023/01/23 16:00:22 GoVersion: 1.19.2
2023/01/23 16:00:22 Platform: linux
2023/01/23 16:00:22 Constraint_parser: >= 1.0, <= 2.0
2023/01/23 16:00:22 Constraint_scenario: >= 1.0, < 3.0
2023/01/23 16:00:22 Constraint_api: v1
2023/01/23 16:00:22 Constraint_acquis: >= 1.0, < 2.0

The bouncer:

2023/01/23 16:01:01 version: v1.4.5-debian-pragmatic-a9a2186a76af63551352aa3bc296bdbe80ca4893
2023/01/23 16:01:01 Codename: alphaga
2023/01/23 16:01:01 BuildDate: 2023-01-19_15:06:57
2023/01/23 16:01:01 GoVersion: 1.19.2
2023/01/23 16:01:01 Platform: linux
2023/01/23 16:01:01 Constraint_parser: >= 1.0, <= 2.0
2023/01/23 16:01:01 Constraint_scenario: >= 1.0, < 3.0
2023/01/23 16:01:01 Constraint_api: v1
2023/01/23 16:01:01 Constraint_acquis: >= 1.0, < 2.0

OS version

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux fully 4.19.0-20-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64 GNU/Linux

Enabled collections and parsers

$ cscli hub list -o raw
crowdsecurity/apache2,enabled,0.1,apache2 support : parser and generic http scenarios ,collections
crowdsecurity/base-http-scenarios,"enabled,update-available",0.6,http common : scanners detection,collections
crowdsecurity/dovecot,enabled,0.1,dovecot support : parser and spammer detection,collections
crowdsecurity/http-cve,"enabled,update-available",1.7,,collections
crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/postfix,enabled,0.2,postfix support : parser and spammer detection,collections
crowdsecurity/smb,enabled,0.1,smb support : parser and brute-force scenario,collections
crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections
crowdsecurity/apache2-logs,enabled,1.3,Parse Apache2 access and error logs,parsers
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/dovecot-logs,enabled,0.4,Parse dovecot logs,parsers
crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/postfix-logs,enabled,0.4,Parse postfix logs,parsers
crowdsecurity/postscreen-logs,enabled,0.2,Parse postscreen logs,parsers
crowdsecurity/smb-logs,enabled,0.2,Parse SMB logs,parsers
crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers
crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios
crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios
crowdsecurity/dovecot-spam,enabled,0.3,detect errors on dovecot,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,0.7,Detect bad user-agents,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios
crowdsecurity/http-generic-bf,"enabled,update-available",0.2,Detect generic http brute force,scenarios
crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios
crowdsecurity/postfix-spam,enabled,0.2,Detect spammers,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios
crowdsecurity/smb-bf,enabled,0.1,Detect smb bruteforce,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios
ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios

Acquisition config

filenames:
  - /var/log/apache2/www.zzz.net.au_error_ssl.log
  - /var/log/apache2/www.zzz.net.au_access_ssl.log
  - /var/log/apache2/other_vhosts_access.log
  - /var/log/apache2/error.log
labels:
  type: apache2
---
#Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log
filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: smb) / files :
journalctl_filter:
 - _SYSTEMD_UNIT=smb.service
labels:
  type: smb
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/messages
filenames:
  - /var/log/syslog
  - /var/log/messages
labels:
  type: syslog
---

</details>


### Config show

<details>

```console
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
  - Hub Folder              : /etc/crowdsec/hub
Local API Server:
  - Listen URL              : :8080
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

Acquisition Metrics:
+-------------------------------------------------------+------------+--------------+----------------+------------------------+
|                        Source                         | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket |
+-------------------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/apache2/other_vhosts_access.log         | 142        | 142          | -              | 41                     |
| file:/var/log/apache2/www.zzz.net.au_access_ssl.log | 140        | 140          | -              | 163                    |
| file:/var/log/apache2/www.zzz.net.au_error_ssl.log  | 19         | 2            | 17             | -                      |
| file:/var/log/auth.log                                | 694        | 3            | 691            | 8                      |
| file:/var/log/syslog                                  | 3.96k      | 552          | 3.41k          | 11                     |
| journalctl:journalctl-_SYSTEMD_UNIT=smb.service       | 1          | -            | 1              | -                      |
+-------------------------------------------------------+------------+--------------+----------------+------------------------+

Bucket Metrics:
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                Bucket                | Current Count | Overflows | Instantiated | Poured | Expired |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-bad-user-agent    | 1             | 1         | 2            | 3      | -       |
| crowdsecurity/http-crawl-non_statics | -             | -         | 48           | 75     | 48      |
| crowdsecurity/http-probing           | 1             | 10        | 19           | 123    | 8       |
| crowdsecurity/http-sensitive-files   | -             | -         | 2            | 3      | 2       |
| crowdsecurity/postfix-spam           | -             | -         | 10           | 11     | 10      |
| crowdsecurity/postscreen-rbl         | -             | 40        | 40           | -      | -       |
| crowdsecurity/ssh-bf                 | -             | -         | 1            | 3      | 1       |
| crowdsecurity/ssh-bf_user-enum       | -             | -         | 1            | 1      | 1       |
| crowdsecurity/ssh-slow-bf            | 1             | -         | 1            | 3      | -       |
| crowdsecurity/ssh-slow-bf_user-enum  | 1             | -         | 1            | 1      | -       |
+--------------------------------------+---------------+-----------+--------------+--------+---------+

Parser Metrics:
+----------------------------------------+-------+--------+----------+
|                Parsers                 | Hits  | Parsed | Unparsed |
+----------------------------------------+-------+--------+----------+
| child-child-crowdsecurity/apache2-logs | 1     | 1      | -        |
| child-crowdsecurity/apache2-logs       | 320   | 284    | 36       |
| child-crowdsecurity/dovecot-logs       | 1.37k | 501    | 870      |
| child-crowdsecurity/http-logs          | 852   | 769    | 83       |
| child-crowdsecurity/postfix-logs       | 3.29k | 11     | 3.28k    |
| child-crowdsecurity/postscreen-logs    | 1.04k | 40     | 997      |
| child-crowdsecurity/smb-logs           | 2     | -      | 2        |
| child-crowdsecurity/sshd-logs          | 4.32k | 3      | 4.32k    |
| child-crowdsecurity/syslog-logs        | 4.65k | 4.65k  | -        |
| crowdsecurity/apache2-logs             | 301   | 284    | 17       |
| crowdsecurity/dateparse-enrich         | 839   | 839    | -        |
| crowdsecurity/dovecot-logs             | 791   | 501    | 290      |
| crowdsecurity/geoip-enrich             | 839   | 839    | -        |
| crowdsecurity/http-logs                | 284   | 281    | 3        |
| crowdsecurity/non-syslog               | 302   | 302    | -        |
| crowdsecurity/postfix-logs             | 1.10k | 11     | 1.09k    |
| crowdsecurity/postscreen-logs          | 1.04k | 40     | 997      |
| crowdsecurity/smb-logs                 | 1     | -      | 1        |
| crowdsecurity/sshd-logs                | 434   | 3      | 431      |
| crowdsecurity/syslog-logs              | 4.65k | 4.65k  | -        |
| crowdsecurity/whitelists               | 839   | 839    | -        |
+----------------------------------------+-------+--------+----------+

Local Api Metrics:
+----------------------+--------+------+
|        Route         | Method | Hits |
+----------------------+--------+------+
| /v1/alerts           | GET    | 5    |
| /v1/alerts           | POST   | 42   |
| /v1/decisions/stream | GET    | 2374 |
| /v1/heartbeat        | GET    | 793  |
| /v1/watchers/login   | POST   | 21   |
+----------------------+--------+------+

Local Api Machines Metrics:
+--------------------------------------------------+---------------+--------+------+
|                     Machine                      |     Route     | Method | Hits |
+--------------------------------------------------+---------------+--------+------+
| 23958888ea41f66471887f73518cf0d3PA1ByO7yLUlwE7de | /v1/alerts    | GET    | 1    |
| 23958888ea41f66471887f73518cf0d3PA1ByO7yLUlwE7de | /v1/heartbeat | GET    | 394  |
| 90d76ef7a42dfbacfc4ab5c054269104lpP7VCRX0SBMGx5q | /v1/alerts    | POST   | 42   |
| 90d76ef7a42dfbacfc4ab5c054269104lpP7VCRX0SBMGx5q | /v1/alerts    | GET    | 4    |
| 90d76ef7a42dfbacfc4ab5c054269104lpP7VCRX0SBMGx5q | /v1/heartbeat | GET    | 398  |
+--------------------------------------------------+---------------+--------+------+

Local Api Bouncers Metrics:
+---------+----------------------+--------+------+
| Bouncer |        Route         | Method | Hits |
+---------+----------------------+--------+------+
| bastion | /v1/decisions/stream | GET    | 2374 |
+---------+----------------------+--------+------+

Local Api Decisions:
+--------------------------------------------+----------+--------+-------+
|                   Reason                   |  Origin  | Action | Count |
+--------------------------------------------+----------+--------+-------+
| crowdsecurity/postscreen-rbl               | crowdsec | ban    | 23    |
| crowdsecurity/f5-big-ip-cve-2020-5902      | CAPI     | ban    | 2     |
| crowdsecurity/http-backdoors-attempts      | CAPI     | ban    | 112   |
| crowdsecurity/http-cve-2021-41773          | CAPI     | ban    | 53    |
| crowdsecurity/postfix-spam                 | CAPI     | ban    | 926   |
| crowdsecurity/spring4shell_cve-2022-22965  | CAPI     | ban    | 2     |
| crowdsecurity/http-probing                 | CAPI     | ban    | 3259  |
| crowdsecurity/http-probing                 | crowdsec | ban    | 1     |
| crowdsecurity/ssh-bf                       | CAPI     | ban    | 13464 |
| crowdsecurity/vmware-cve-2022-22954        | CAPI     | ban    | 2     |
| crowdsecurity/dovecot-spam                 | CAPI     | ban    | 246   |
| crowdsecurity/http-bad-user-agent          | CAPI     | ban    | 3993  |
| crowdsecurity/http-bad-user-agent          | crowdsec | ban    | 1     |
| crowdsecurity/http-crawl-non_statics       | CAPI     | ban    | 450   |
| crowdsecurity/http-generic-bf              | CAPI     | ban    | 7     |
| crowdsecurity/http-path-traversal-probing  | CAPI     | ban    | 117   |
| crowdsecurity/fortinet-cve-2018-13379      | CAPI     | ban    | 31    |
| crowdsecurity/http-open-proxy              | CAPI     | ban    | 132   |
| crowdsecurity/jira_cve-2021-26086          | CAPI     | ban    | 35    |
| crowdsecurity/thinkphp-cve-2018-20062      | CAPI     | ban    | 36    |
| ltsich/http-w00tw00t                       | CAPI     | ban    | 4     |
| crowdsecurity/ssh-slow-bf                  | CAPI     | ban    | 3777  |
| crowdsecurity/CVE-2022-26134               | CAPI     | ban    | 1     |
| crowdsecurity/apache_log4j2_cve-2021-44228 | CAPI     | ban    | 16    |
| crowdsecurity/http-cve-2021-42013          | CAPI     | ban    | 1     |
| crowdsecurity/http-sensitive-files         | CAPI     | ban    | 120   |
| crowdsecurity/smb-bf                       | CAPI     | ban    | 1914  |
+--------------------------------------------+----------+--------+-------+

Local Api Alerts:
+---------------------------------------+-------+
|                Reason                 | Count |
+---------------------------------------+-------+
| crowdsecurity/thinkphp-cve-2018-20062 | 1     |
| crowdsecurity/CVE-2022-41082          | 5     |
| crowdsecurity/http-bad-user-agent     | 49    |
| crowdsecurity/http-crawl-non_statics  | 1     |
| crowdsecurity/http-probing            | 6     |
| crowdsecurity/http-sensitive-files    | 1     |
| crowdsecurity/jira_cve-2021-26086     | 2     |
| crowdsecurity/postscreen-rbl          | 913   |
+---------------------------------------+-------+

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

None.

lingfish avatar Jan 23 '23 05:01 lingfish

@lingfish: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

github-actions[bot] avatar Jan 23 '23 05:01 github-actions[bot]

So I prewarn calling the api using the same key as the bouncer will cause issue as the way the system works is it calculates the time different between calls and that how it knows what is new and old.

I will try to see to reproduce but there is not much to go on here. Is the bouncer actually working? Have you tested by triggering an alert then seeing if you can still gain access?

LaurenceJJones avatar Jan 23 '23 10:01 LaurenceJJones

Hi. I only started poking into the API using curl after the issue started, so I'm pretty sure what I did hasn't caused it; I understand your warning though going forward.

I get it's a strange one, but if there's any logs etc I can turn up to trace, let me know. I know there's not much to go on, need your guidance.

The bouncer is otherwise working just fine; bans are installed into a nftables set etc... it's just this one single IP. I don't think the issue is the bouncer, as it's never being told about this weird IP, only others. An example of a addition:

time="22-01-2023 15:04:00" level=debug msg="req-api: GET http://<LAPI host>:8080/v1/decisions/stream?origins=cscli%2Ccrowdsec"
time="22-01-2023 15:04:00" level=trace msg="auth-api request: GET /v1/decisions/stream?origins=cscli%2Ccrowdsec HTTP/1.1\r\nHost: <LAPI host>:8080\r\nUser-Agent: crowdsec-firewall-bouncer/v0.0.24-debian-pragmatic-<long UUID>\r\nX-Api-Key: <API key>\r\n\r\n"
time="22-01-2023 15:04:00" level=trace msg="auth-api response: HTTP/1.1 200 OK\r\nContent-Length: 184\r\nContent-Type: application/json; charset=utf-8\r\nDate: Sun, 22 Jan 2023 04:04:00 GMT\r\n\r\n{\"deleted\":null,\"new\":[{\"duration\":\"3h59m58.296214104s\",\"id\":3620822,\"origin\":\"crowdsec\",\"scenario\":\"crowdsecurity/postscreen-rbl\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"185.225.73.170\"}]}"
time="22-01-2023 15:04:00" level=debug msg="resp-api: http 200"
time="22-01-2023 15:04:00" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="22-01-2023 15:04:00" level=debug msg="[headers] Date : [Sun, 22 Jan 2023 04:04:00 GMT]"
time="22-01-2023 15:04:00" level=debug msg="[headers] Content-Length : [184]"
time="22-01-2023 15:04:00" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 184\r\nContent-Type: application/json; charset=utf-8\r\nDate: Sun, 22 Jan 2023 04:04:00 GMT\r\n\r\n{\"deleted\":null,\"new\":[{\"duration\":\"3h59m58.296214104s\",\"id\":3620822,\"origin\":\"crowdsec\",\"scenario\":\"crowdsecurity/postscreen-rbl\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"185.225.73.170\"}]}"
time="22-01-2023 15:04:00" level=debug msg="Adding '185.225.73.170' for '3h59m58.296214104s'"
time="22-01-2023 15:04:00" level=debug msg="committing added decisions"
time="22-01-2023 15:04:00" level=debug msg="adding 185.225.73.170 to buffer "
time="22-01-2023 15:04:00" level=debug msg="committed added decisions"
time="22-01-2023 15:04:00" level=info msg="1 decision added"

When the bouncer starts, it sees other IPs already in a decision, just not the magical IP:

time="22-01-2023 14:28:40" level=debug msg="Adding '185.225.73.170' for '35m1.949202109s'"
time="22-01-2023 14:28:40" level=debug msg="Adding '2.57.122.215' for '1h16m13.776833835s'"
time="22-01-2023 14:28:40" level=debug msg="Adding '162.142.125.213' for '3h15m18.263834884s'"
time="22-01-2023 14:28:40" level=debug msg="committing added decisions"
time="22-01-2023 14:28:40" level=debug msg="adding 162.142.125.213 to buffer "
time="22-01-2023 14:28:40" level=debug msg="adding 185.225.73.170 to buffer "
time="22-01-2023 14:28:40" level=debug msg="adding 2.57.122.215 to buffer "
time="22-01-2023 14:28:40" level=debug msg="committed added decisions"
time="22-01-2023 14:28:40" level=info msg="3 decisions added"

The LAPI notices this weird IP, logs about it, but strangely never gets reported to the bouncer:

time="24-01-2023 09:22:15" level=info msg="Bucket overflow" bucket_id=hidden-shadow capacity=0 cfg=hidden-hill file=/etc/crowdsec/scenarios/postfix-spam.yaml name=crowdsecurity/postscreen-rbl partition=1ebdee2c8992678b607ef9fae0516d5b0352680b
time="24-01-2023 09:22:15" level=info msg="Ip 193.56.29.178 performed 'crowdsecurity/postscreen-rbl' (1 events over 91ns) at 2023-01-23 22:22:15.463192835 +0000 UTC"
time="24-01-2023 09:22:15" level=info msg="(xxx/crowdsec) crowdsecurity/postscreen-rbl by ip 193.56.29.178 (PL/210228) : 4h ban on Ip 193.56.29.178"
time="24-01-2023 09:22:16" level=info msg="sent email to [xxx]" @module=email-plugin.email_default
time="24-01-2023 09:22:25" level=info msg="Signal push: 1 signals to push"
time="24-01-2023 09:23:44" level=info msg="flushed 1/1058 alerts because they were created 7d ago or more"

Due to this, the nftables rule never gets installed, the spammer keeps trying to spam, I keep getting notifications.

lingfish avatar Jan 23 '23 22:01 lingfish

The plot thickens... I have now at least one other IP doing the same thing:

image

lingfish avatar Jan 24 '23 23:01 lingfish

Could there be some sort of bypass they are doing? as going to be straight up and say we cant really replicate without steps....

LaurenceJJones avatar Jan 26 '23 14:01 LaurenceJJones

Could there be some sort of bypass they are doing?

No. The bouncer, running on my boundary firewall, should be blocking them (and hence no further RBL stuff for 4 hours), and it isn't, because the decisions aren't showing up in the API, as posted above.

we cant really replicate without steps....

I understand, but surely we can do some debugging? Logging? Replicate with a dump of my DB?

lingfish avatar Jan 27 '23 01:01 lingfish

You pretty much have all the logging on in the previous comment, so those IP's are most likely in our community blocklist so the filter you are applying for debugging could be not returning it cause the will have the source CAPI.

Is the bouncer running in ipset mode only? I would for debugging create an api key, send the same request with no filters with startup true (send all current IP's) then grep the output for those if they are there, then some reason the LAPI already thinks the bouncer should know these.

LaurenceJJones avatar Jan 27 '23 08:01 LaurenceJJones

so the filter you are applying for debugging could be not returning it cause the will have the source CAPI

I think you've nailed it; I'm using those filters because yes, I've set origins: ["cscli", "crowdsec"] in the bouncer config.

Is the bouncer running in ipset mode only?

Yep, sure is.

I would for debugging create an api key, send the same request with no filters with startup true

Yep, done, and you're right... both IPs are in the list, origin CAPI.

So, being that origins is relatively new, I guess this is a kind of "feature" bug maybe? I would expect that the bouncer would be told to ban, if one intentionally wasn't using the CAPI list.

lingfish avatar Jan 28 '23 06:01 lingfish

I understand issue clearly now. I will need to have a chat with @buixor and see if I can replicate easily without CAPI I can spoof the source but will reply once I get some progress.

LaurenceJJones avatar Jan 30 '23 13:01 LaurenceJJones

Any news here?

lingfish avatar Mar 14 '23 04:03 lingfish

Just to keep you in the loop. We have tested a potential fix, however, it seems the way ent is converting the syntax to a sql statement means it takes minutes to returns a response. So we need to investigate it further.

LaurenceJJones avatar Mar 15 '23 11:03 LaurenceJJones

Hi, has this been resolved yet?

lingfish avatar Jul 17 '23 23:07 lingfish

No, as of yet we haven't found an optimal solution plus we are classing it as an edge case that a user wants to supply signals to CAPI but not use the community blocklist within a remediation component. So we have tagged this a low priority since we got multiple internal projects that we want to finish firstly.

LaurenceJJones avatar Jul 18 '23 12:07 LaurenceJJones

The only idea I had to workaround until a full feature fix is implemented is add additional config to CAPI configuration to allow you to set a flag to "send only"

Example

url: https://api.crowdsec.net/
login: XXXX
password: XXXX
share_only : true  ## setting to true will cancel pull tomb

Then we create if statement for pull tomb

https://github.com/crowdsecurity/crowdsec/blob/bb16552aca20b71b90cfecc4145acdf3924d1438/pkg/apiserver/apiserver.go#L350-L356

Proof of concept #2362

LaurenceJJones avatar Jul 18 '23 12:07 LaurenceJJones