CrowdSec plugin on OpnSense not starting as expected

[markoverholser]

What happened?

After installing the crowdsec plugin from the OpnSense plugin manager, listed as version 1.0.6 at the time of attempt, I get the following log below. (In particular, note the messages saying Cannot 'start' <service>. Set <service>_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.) Even after toggling the LAPI off and back on again from Services > Crowdsec > Settings and applying the settings change, the services are not started, the LAPI is not listening on the port (as verified by sockstat -4 -l | grep 8080).

I am able to manually run the LAPI binary with crowdsec -c /usr/local/etc/crowdsec/config.yaml and it starts, then I can see that it is listening on 8080 and I am able to interact with the LAPI using cscli. I am also able to issue the command rctl crowdsec onestart and start the service. If I do start the service manually, disabling the LAPI in the settings page and applying does not stop the LAPI service I enabled manually, so it appears whatever mechanism that is supposed to be enabling/disabling/starting/stopping the services is not working as expected.

Additionally, when I was testing and troubleshooting, initially my cscli would not interact with the LAPI until I executed cscli machines add -a at the command-line. I was not expecting (based on the documentation) to have to do that, either.

Below is the full installation log from my attempt, in case it helps any. Please let me know if there is something I have done incorrectly, or if you want me to aid in troubleshooting.

***GOT REQUEST TO INSTALL***
Currently running OPNsense 23.1.11_1 at Thu Aug 24 09:21:42 CDT 2023
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	crowdsec: 1.5.1
	crowdsec-firewall-bouncer: 0.0.27
	os-crowdsec: 1.0.6

Number of packages to be installed: 3

The process will require 144 MiB more space.
[1/3] Installing crowdsec-firewall-bouncer-0.0.27...
[1/3] Extracting crowdsec-firewall-bouncer-0.0.27: ...... done
Cannot 'start' crowdsec_firewall. Set crowdsec_firewall_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
[2/3] Installing crowdsec-1.5.1...
[2/3] Extracting crowdsec-1.5.1: .......... done
Cannot 'start' crowdsec. Set crowdsec_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
[3/3] Installing os-crowdsec-1.0.6...
[3/3] Extracting os-crowdsec-1.0.6: .......... done
Stopping configd...done
Starting configd.
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/CrowdSec: OK
OK
=====
Message from crowdsec-firewall-bouncer-0.0.27:

--
crowdsec-firewall-bouncer is installed.

If you are running crowdsec on this machine, the bouncer will register itself with
the Local API when it's started the first time.

If the LAPI is on another machine, you need to manually register the bouncer
and fill api_key and api_url in /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml before
starting the service.

This package depends on the Packet Filter service.
To make sure it's active:

----------
# sysrc pf_enable=YES
pf_enable: NO -> YES
# service pf start
Enabling pf.
----------

Add the following in /etc/pf.conf to create the firewall tables and rules:

----------
table <crowdsec-blacklists> persist
table <crowdsec6-blacklists> persist
block drop in quick from <crowdsec-blacklists> to any
block drop in quick from <crowdsec6-blacklists> to any
----------

To apply the file:

# pfctl -f /etc/pf.conf

Then activate the bouncer via sysrc and run it:

----------
# sysrc crowdsec_firewall_enable="YES"
crowdsec_firewall_enable: NO -> YES
# service crowdsec_firewall start
----------
=====
Message from crowdsec-1.5.1:

--
crowdsec is installed.

You need to check/edit the following files in /usr/local/etc/crowdsec as described in https://doc.crowdsec.net/docs/configuration/crowdsec_configuration

 - config.yaml: main configuration
 - acquis.yaml, acquis.d: datasource configuration (this port does not include automatic discovery of the running services)
 - profiles.yaml: remediation policies (ban, duration, etc)

Then you can enable the daemon via sysrc and run it.

# sysrc crowdsec_enable="YES"
crowdsec_enable: NO -> YES
# service crowdsec start
Checking integrity... done (0 conflicting)
Nothing to do.
***DONE***

What did you expect to happen?

I expected crowdsec to install, and for the services to start/stop according to the checkboxes in the Settings page.

How can we reproduce it (as minimally and precisely as possible)?

1. Obtain an OpnSense installation
2. Install the opnsense package
3. Enable the LAPI via Services > CrowdSec > Settings and Apply
4. Use sockstat -l -4 | grep <LAPI port> to verify the LAPI is not listening

Anything else we need to know?

No response

Crowdsec version

2023/08/24 09:45:06 version: v1.5.1-freebsd-b76e95e3
2023/08/24 09:45:06 Codename: alphaga
2023/08/24 09:45:06 BuildDate: 2023-06-27_02:17:19
2023/08/24 09:45:06 GoVersion: 1.20.3
2023/08/24 09:45:06 Platform: freebsd
2023/08/24 09:45:06 Constraint_parser: >= 1.0, <= 2.0
2023/08/24 09:45:06 Constraint_scenario: >= 1.0, < 3.0
2023/08/24 09:45:06 Constraint_api: v1
2023/08/24 09:45:06 Constraint_acquis: >= 1.0, < 2.0

OS version

FreeBSD bleu.as-local 13.1-RELEASE-p8 FreeBSD 13.1-RELEASE-p8 stable/23.1-n250461-cf0d42d1ca7 SMP amd64

Enabled collections and parsers

No response

Acquisition config

No response

Config show

No response

Prometheus metrics

No response

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

No response

[github-actions[bot]]

@markoverholser: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

Can you provide the logs from when the service failed to start /usr/local/var/log/crowdsec/ <- i think this is the location on freebsd

But check /var/log/ if im wrong

[LaurenceJJones]

I believe the LAPI section you speak of is for a remote lapi and enabling this does turn of the local API. Maybe @mmetc can explain as I never used the opnsense plugin

[markoverholser]

Can you provide the logs from when the service failed to start /usr/local/var/log/crowdsec/ <- i think this is the location on freebsd

But check /var/log/ if im wrong

I removed /var/log/crowdsec/ and /usr/local/etc/crowdsec/ and reinstalled the plugin to start from scratch again.

1. Logs are in /var/log/crowdsec/
2. In there, only crowdsec-firewall-bouncer.log exists
3. Here are the contents of crowdsec-firewall-bouncer.log

time="24-08-2023 11:48:06" level=info msg="backend type : pf"
time="24-08-2023 11:48:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec_blacklists -T flush"
time="24-08-2023 11:48:06" level=info msg="Checking pf table: crowdsec_blacklists"
time="24-08-2023 11:48:06" level=info msg="pf initiated for ipv4"
time="24-08-2023 11:48:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec6_blacklists -T flush"
time="24-08-2023 11:48:06" level=info msg="Checking pf table: crowdsec6_blacklists"
time="24-08-2023 11:48:06" level=info msg="pf initiated for ipv6"
time="24-08-2023 11:48:06" level=info msg="Using API key auth"
time="24-08-2023 11:48:06" level=info msg="Processing new and deleted decisions . . ."
time="24-08-2023 11:48:06" level=info msg="Serving metrics at 127.0.0.1:60601/metrics"
time="24-08-2023 11:48:36" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 172.16.0.1:8080: i/o timeout"
time="24-08-2023 11:48:36" level=error msg="Get \"http://172.16.0.1:8080/v1/decisions/stream?startup=true\": dial tcp 172.16.0.1:8080: i/o timeout"
time="24-08-2023 11:48:36" level=info msg="Shutting down backend"
time="24-08-2023 11:48:36" level=info msg="flushing 'crowdsec' table(s)"
time="24-08-2023 11:48:36" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec_blacklists -T flush"
time="24-08-2023 11:48:36" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec6_blacklists -T flush"
time="24-08-2023 11:48:36" level=fatal msg="process terminated with error: stream api init failed"

I believe the LAPI section you speak of is for a remote lapi and enabling this does turn of the local API. Maybe @mmetc can explain as I never used the opnsense plugin

The help section of the UI for the LAPI option says

Enable/disable the CrowdSec Local API. Keep this enabled unless you connect to a LAPI on another machine.

which contradicts your statement.

I toggled "Enable LAPI" off (it is on by default) and Applied the changes, there was no change in behavior. Nothing listening on 8080, and no new logs in /var/log/crowdsec/. I toggled it back on, and there was a repeat of the above behavior in /var/log/crowdsec/crowdsec-firewall-bouncer.log but no new logs for the LAPI service, nothing listening on 8080, running cscli decisions list just times out, etc.