crowdsec crashing caused by corrupt journald files

[BPplays]

What happened?

crowdsec exits with this error when there is a corrupted journald file

 2月 20 10:22:04 censored crowdsec[1631985]: FATAL unable to start crowdsec routines: starting acquisition error: journalctl error : Journal file /var/log/journal/a9c76294199a4f88adf443b4c9e7ddf6/user-1000@875e8bb697bc464dad39bdc08199af20-0000000000000000-0000000000000000.journal corrupted, ignoring file.

What did you expect to happen?

journald seems to handle corrupted files fine so there doesn't seem to be a reason to crash in this case see: this

How can we reproduce it (as minimally and precisely as possible)?

1. have corrupted journald file, check with journalctl --verify
2. try to run crowdsec

Anything else we need to know?

No response

Crowdsec version

$ cscli version
version: v1.6.5-debian-pragmatic-amd64-d8dcdc91
Codename: alphaga
BuildDate: 2025-02-07_14:53:23
GoVersion: 1.23.6
Platform: linux
libre2: C++
User-Agent: crowdsec/v1.6.5-debian-pragmatic-amd64-d8dcdc91-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux dns1.suzuko.org 6.1.0-29-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.123-1 (2025-01-02) x86_64 GNU/Linux

Enabled collections and parsers

$ cscli hub list -o raw
# paste output here

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here

On Windows:

C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml

paste output here

Config show

$ cscli config show
# paste output here

Prometheus metrics

$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions[bot]]

@BPplays: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

So can you confirm that journalctl is the only acquistion you have? as the fatal should only happen if you only have one acquisition and we fail to spin up the only one it has configured?

debugging stuff:

So we attach a scanner to stderr and if a single line is sent to stderr then we class it as a reason to kill

https://github.com/crowdsecurity/crowdsec/blob/0bdb1f7f27a0c07095a4ce93998f89050ba54f05/pkg/acquisition/modules/journalctl/journalctl.go#L153-L156

I dont know what journalctl other than corrupt files would send to stderr and if its a reasonable thing to guess we should kill it?

[BPplays]

@LaurenceJJones journalctl is not the only acquistion i have

-> cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
#Generated acquisition file - wizard.sh (service: ssh) / files : /var/log/auth.log
filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: smb) / files :
journalctl_filter:
 - _SYSTEMD_UNIT=smb.service
labels:
  type: smb
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log
filenames:
  - /var/log/syslog
  - /var/log/kern.log
labels:
  type: syslog
---
---
source: docker
container_name:
 - apache_guacamole
labels:
  type: apache-guacamole
---
source: docker
container_name:
 - authentik_server
labels:
  type: authentik

here is the specific journalctl error if you want to try to only filter it out:

Journal file /var/log/journal/a9c76294199a4f88adf443b4c9e7ddf6/user-1000@875e8bb697bc464dad39bdc08199af20-0000000000000000-0000000000000000.journal corrupted, ignoring file.

here is my full journalctl logs for crowdsec (2>/dev/null filters out the above message)

-> sudo journalctl -eu crowdsec -o short-unix 2>/dev/null | cat
1740493736.430497 dns1.domain systemd[1]: Reloading crowdsec.service - Crowdsec agent...
1740493736.599678 dns1.domain crowdsec[3499520]: time="2025-02-25T06:28:56-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740493738.713793 dns1.domain systemd[1]: Reloaded crowdsec.service - Crowdsec agent.
1740686211.380451 dns1.domain systemd[1]: Stopping crowdsec.service - Crowdsec agent...
1740686214.492942 dns1.domain systemd[1]: crowdsec.service: Deactivated successfully.
1740686214.493497 dns1.domain systemd[1]: Stopped crowdsec.service - Crowdsec agent.
1740686214.493576 dns1.domain systemd[1]: crowdsec.service: Consumed 24min 2.368s CPU time.
1740686214.530839 dns1.domain systemd[1]: Starting crowdsec.service - Crowdsec agent...
1740686214.724782 dns1.domain crowdsec[91442]: time="2025-02-27T11:56:54-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686217.179368 dns1.domain crowdsec[91482]: time="2025-02-27T11:56:57-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686219.203193 dns1.domain systemd[1]: Started crowdsec.service - Crowdsec agent.
1740686219.343322 dns1.domain crowdsec[91482]: FATAL unable to start crowdsec routines: starting acquisition error: journalctl error : Journal file /var/log/journal/a9c76294199a4f88adf443b4c9e7ddf6/user-1000@875e8bb697bc464dad39bdc08199af20-0000000000000000-0000000000000000.journal corrupted, ignoring file.
1740686219.350488 dns1.domain systemd[1]: crowdsec.service: Main process exited, code=exited, status=1/FAILURE
1740686219.350745 dns1.domain systemd[1]: crowdsec.service: Failed with result 'exit-code'.
1740686219.351049 dns1.domain systemd[1]: crowdsec.service: Consumed 2.910s CPU time.
1740686279.531333 dns1.domain systemd[1]: crowdsec.service: Scheduled restart job, restart counter is at 1.
1740686279.531567 dns1.domain systemd[1]: Stopped crowdsec.service - Crowdsec agent.
1740686279.531607 dns1.domain systemd[1]: crowdsec.service: Consumed 2.910s CPU time.
1740686279.562722 dns1.domain systemd[1]: Starting crowdsec.service - Crowdsec agent...
1740686279.604108 dns1.domain crowdsec[91873]: time="2025-02-27T11:57:59-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686281.655258 dns1.domain crowdsec[92023]: time="2025-02-27T11:58:01-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686283.578972 dns1.domain systemd[1]: Started crowdsec.service - Crowdsec agent.
1740686283.654928 dns1.domain crowdsec[92023]: FATAL unable to start crowdsec routines: starting acquisition error: journalctl error : Journal file /var/log/journal/a9c76294199a4f88adf443b4c9e7ddf6/user-1000@875e8bb697bc464dad39bdc08199af20-0000000000000000-0000000000000000.journal corrupted, ignoring file.
1740686283.663040 dns1.domain systemd[1]: crowdsec.service: Main process exited, code=exited, status=1/FAILURE
1740686283.663220 dns1.domain systemd[1]: crowdsec.service: Failed with result 'exit-code'.
1740686283.663585 dns1.domain systemd[1]: crowdsec.service: Consumed 2.948s CPU time.
1740686343.781145 dns1.domain systemd[1]: crowdsec.service: Scheduled restart job, restart counter is at 2.
1740686343.781382 dns1.domain systemd[1]: Stopped crowdsec.service - Crowdsec agent.
1740686343.781420 dns1.domain systemd[1]: crowdsec.service: Consumed 2.948s CPU time.
1740686343.811208 dns1.domain systemd[1]: Starting crowdsec.service - Crowdsec agent...
1740686343.856190 dns1.domain crowdsec[92517]: time="2025-02-27T11:59:03-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686345.930035 dns1.domain crowdsec[92543]: time="2025-02-27T11:59:05-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686347.952816 dns1.domain systemd[1]: Started crowdsec.service - Crowdsec agent.
1740686348.035595 dns1.domain crowdsec[92543]: FATAL unable to start crowdsec routines: starting acquisition error: journalctl error : Journal file /var/log/journal/a9c76294199a4f88adf443b4c9e7ddf6/user-1000@875e8bb697bc464dad39bdc08199af20-0000000000000000-0000000000000000.journal corrupted, ignoring file.
1740686348.042031 dns1.domain systemd[1]: crowdsec.service: Main process exited, code=exited, status=1/FAILURE
1740686348.042213 dns1.domain systemd[1]: crowdsec.service: Failed with result 'exit-code'.
1740686348.042561 dns1.domain systemd[1]: crowdsec.service: Consumed 2.551s CPU time.
1740686408.280791 dns1.domain systemd[1]: crowdsec.service: Scheduled restart job, restart counter is at 3.
1740686408.281092 dns1.domain systemd[1]: Stopped crowdsec.service - Crowdsec agent.
1740686408.281139 dns1.domain systemd[1]: crowdsec.service: Consumed 2.551s CPU time.
1740686408.302496 dns1.domain systemd[1]: Starting crowdsec.service - Crowdsec agent...
1740686408.352598 dns1.domain crowdsec[93520]: time="2025-02-27T12:00:08-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686410.438583 dns1.domain crowdsec[93548]: time="2025-02-27T12:00:10-08:00" level=info msg="Loading yaml file: '/etc/crowdsec/config.yaml' with additional values from '/etc/crowdsec/config.yaml.local'"
1740686412.490772 dns1.domain systemd[1]: Started crowdsec.service - Crowdsec agent.
1740686412.574549 dns1.domain crowdsec[93548]: FATAL unable to start crowdsec routines: starting acquisition error: journalctl error : Journal file /var/log/journal/a9c76294199a4f88adf443b4c9e7ddf6/user-1000@875e8bb697bc464dad39bdc08199af20-0000000000000000-0000000000000000.journal corrupted, ignoring file.
1740686412.581917 dns1.domain systemd[1]: crowdsec.service: Main process exited, code=exited, status=1/FAILURE
1740686412.582096 dns1.domain systemd[1]: crowdsec.service: Failed with result 'exit-code'.
1740686412.582490 dns1.domain systemd[1]: crowdsec.service: Consumed 2.674s CPU time.

[BPplays]

@LaurenceJJones also is using the journalctl command really the best choice. it looks like sd-journal allows reading from journald, there seems to be a golang package