UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')

[C8opmBM]

What happened?

Hello,

I've been using crowdsec for maybe 2 years, it was a a set and forget thing, but lately I've seen constant errors in the logs.

Every 2-3 lines, I get one of these:

UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')

I haven't made any changes to the config.

What did you expect to happen?

No errors

How can we reproduce it (as minimally and precisely as possible)?

Run crowdsec with authelia and some collections

Anything else we need to know?

My docker

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    profiles:
      - backend
    networks:
      - system
    security_opt:
      - no-new-privileges=true
    ports:
      - ${PORT_CROWDSEC_API}:8080 # exposes a REST API for bouncers, cscli and communication between crowdsec agent and local api
      - ${PORT_CROWDSEC_METRICS}:6060 #exposes prometheus metrics on /metrics and pprof debugging metrics on /debug
    environment:
      TZ: ${TZ}
      GID: ${PGID}
      COLLECTIONS: "crowdsecurity/caddy LePresidente/authelia"
      POSTOVERFLOWS: "crowdsecurity/rdns"
      CUSTOM_HOSTNAME: omv
      #DOCKER_HOST: tcp://socky_proxy:${PORT_SOCKY_PROXY} 
    volumes:
      - ${CONFIGDIR}/crowdsec/config/acquis.yaml:/etc/crowdsec/acquis.yaml:ro
      - ${CONFIGDIR}/crowdsec/myconf/mywhitelists.yaml:/etc/crowdsec/postoverflows/s01-whitelists/mywhitelists.yaml:ro
      - ${CONFIGDIR}/crowdsec/config:/etc/crowdsec/
      - ${CONFIGDIR}/crowdsec/data:/var/lib/crowdsec/data/
      - ${LOGDIR}/caddy:/logs/caddy:ro
      - ${LOGDIR}/authelia:/logs/authelia:ro

      #- /var/log/auth.log:/logs/auth.log:ro
      #- /var/log/syslog.log:/logs/syslog.log:ro
    #healthcheck:
    #  test: ["CMD", "cscli", "version"]
    restart: always

Crowdsec version

$ cscli version
2024/02/24 20:30:55 version: v1.6.0-4192af30
2024/02/24 20:30:55 Codename: alphaga
2024/02/24 20:30:55 BuildDate: 2024-01-31_12:35:08
2024/02/24 20:30:55 GoVersion: 1.21.6
2024/02/24 20:30:56 Platform: docker
2024/02/24 20:30:56 libre2: C++
2024/02/24 20:30:56 Constraint_parser: >= 1.0, <= 3.0
2024/02/24 20:30:56 Constraint_scenario: >= 1.0, <= 3.0
2024/02/24 20:30:56 Constraint_api: v1
2024/02/24 20:30:56 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux omv 6.1.0-0.deb11.13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1~bpo11+1 (2023-10-08) x86_64 GNU/Linux

Enabled collections and parsers

$ cscli hub list -o raw
name,status,version,description,type
crowdsecurity/caddy-logs,enabled,0.7,Parse caddy logs,parsers
crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers
crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/sshd-logs,enabled,2.3,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers
LePresidente/authelia-logs,enabled,0.5,Parse Authelia logs,parsers
crowdsecurity/rdns,enabled,0.3,Lookup the DNS associated to the source IP only for overflows,postoverflows
me/rdns_whitelist,"enabled,local",,,postoverflows
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.5,Detect cve-2021-44228 exploitation attemps,scenarios
crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios
crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios
crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios
crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios
crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios
crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios
crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios
crowdsecurity/http-admin-interface-probing,enabled,0.3,Detect generic HTTP admin interface probing,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.5,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,1.1,Detect usage of bad User Agent,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios
crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios
crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.3,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.3,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sensitive-files,enabled,0.3,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.3,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-xss-probing,enabled,0.3,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.2,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios
crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.4,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios
LePresidente/authelia-bf,enabled,0.3,Detect authelia bruteforce,scenarios
ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios
crowdsecurity/bf_base,enabled,0.1,,contexts
crowdsecurity/http_base,enabled,0.2,,contexts
crowdsecurity/base-http-scenarios,enabled,0.8,http common : scanners detection,collections
crowdsecurity/caddy,enabled,0.1,caddy support : parser and generic http scenarios,collections
crowdsecurity/http-cve,enabled,2.5,Detect CVE exploitation in http logs,collections
crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/sshd,enabled,0.3,sshd support : parser and brute-force detection,collections
LePresidente/authelia,enabled,0.2,Authelia Support : parser and brute-force detection,collections

Acquisition config

No response

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : stdout
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : http://0.0.0.0:8080/
  - Login                   : omv
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:8080
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

No response

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions[bot]]

@C8opmBM: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[C8opmBM]

I think it's related to https://github.com/crowdsecurity/hub/pull/961 But I don't know how to fix it.

[LaurenceJJones]

I think it's related to crowdsecurity/hub#961 But I don't know how to fix it.

The reason is seems authelia is in Debug mode and is logging lines that are not captured by the first node so it falls into the Unmarshal json which then generally errors because its a not a JSON line

[C8opmBM]

Unfortunately changing the log level to info (in authelia) does not fix it.

So I thought this was only related to LePresidente/Authelia but I noticed also Unmarshall errors with other unrelated errors/warns. Same when parsing the caddy logs, etc.

LE: I should add that I tested the filter (authelia) and it works as intended - blocked my repeated logins - so functionality doesn't seem to be affected. aFAIK I just would like to know what has recently changed, so I can fix these.

My caddy log format is as follows:

        log {
                output file {$LOG_FILE} {
                        roll_size 24b
                        roll_keep 10
                        roll_keep_for 720h
                }
                format json {
                        time_format wall
                        time_local
                }
                level INFO
        }

Do I need to change anything? It has been working fine (as mentioned) for a couple of years until recently.

[LaurenceJJones]

Unfortunately changing the log level to info (in authelia) does not fix it.

So I thought this was only related to LePresidente/Authelia but I noticed also Unmarshall errors with other unrelated errors/warns. Same when parsing the caddy logs, etc.

LE: I should add that I tested the filter (authelia) and it works as intended - blocked my repeated logins - so functionality doesn't seem to be affected. aFAIK I just would like to know what has recently changed, so I can fix these.

My caddy log format is as follows:

        log {
                output file {$LOG_FILE} {
                        roll_size 24b
                        roll_keep 10
                        roll_keep_for 720h
                }
                format json {
                        time_format wall
                        time_local
                }
                level INFO
        }

Do I need to change anything? It has been working fine (as mentioned) for a couple of years until recently.

No you dont need to change anything It still working fine, it just some log lines dont match the parsers so it errors when it tries to parse them as JSON. We could update the parser to check if the first character is a { to suppress the message

[C8opmBM]

Amazing, thank you!!

[KonVei82]

@LaurenceJJones i think i am having a similar issue with multiple errors in logs.

time="2024-03-19T14:39:24+02:00" level=warning msg="failed to run filter : unexpected end of JSON input (1:42)\n | evt.Parsed.program startsWith 'caddy' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'caddy') in ['', nil]\n | .........................................^" id=still-glitter name=crowdsecurity/caddy-logs stage=s01-parse

time="2024-03-19T14:39:24+02:00" level=error msg="UnmarshalJSON : invalid character ',' looking for beginning of value" line=","server_name":"auth.redacted.pro"}},"bytes_read":0,"user_id":"","duration":0.002094356,"size":0,"status":304,"resp_headers":{"Etag":["8e54fc845814eacc4dfc1639edd222a61621f1f7"],"Cache-Control":["public, max-age=0, must-revalidate"],"Server":["Caddy"],"Date":["Tue, 19 Mar 2024 12:39:24 GMT"]}}" 
time="2024-03-19T14:39:24+02:00" level=warning msg="failed to run filter : invalid character ',' looking for beginning of value (1:42)\n | evt.Parsed.program startsWith 'caddy' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'caddy') in ['', nil]\n | .........................................^" id=still-glitter name=crowdsecurity/caddy-logs stage=s01-parse

Let me know if i should open a new bug report.