hub
hub copied to clipboard
crowdsecurity
Error unexpected end of JSON input when processing traefik logs
Describe the bug Ocassionally a set of errors such as the following appear in the crowdsec log:
crowdsec | time="2025-03-04T18:05:02-07:00" level=error msg="UnmarshalJSON : unexpected end of JSON input" line="{\"ClientAddr\":\"192.168.1.105:56612\",\"ClientHost\":\"192.168.1.105\",\"ClientPort\":\"56612\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":9836,\"DownstreamStatus\":200,\"Duration\":557951410,\"OriginContentSize\":9836,\"OriginDuration\":557207858,\"OriginStatus\":200,\"Overhead\":743552,\"RequestAddr\":\"immich.example.com\",\"RequestContentSize\":0,\"RequestCount\":16027,\"RequestHost\":\"immich.example.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/assets/15accaea-6c75-4168-a470-9af475f9a8d3/thumbnail?size=thumbnail\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"id-services-immich-rtr@file\",\"ServiceAddr\":\"192.168.1.93:2283\",\"ServiceName\":\"id-services-immich-srv@file\",\"ServiceURL\":\"http://192.168.1.93:2283\",\"SpanId\":\"0000000000000000\",\"StartLocal\":\"2025-03-04T18:05:02.356967516-"
crowdsec | time="2025-03-04T18:05:02-07:00" level=warning msg="failed to run filter : unexpected end of JSON input (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=lingering-sun name=child-crowdsecurity/traefik-logs stage=s01-parse
crowdsec | time="2025-03-04T18:05:02-07:00" level=error msg="UnmarshalJSON : invalid character '7' after top-level value" line="07:00\",\"StartUTC\":\"2025-03-05T01:05:02.356967516Z\",\"TLSCipher\":\"TLS_CHACHA20_POLY1305_SHA256\",\"TLSVersion\":\"1.3\",\"TraceId\":\"00000000000000000000000000000000\",\"entryPointName\":\"websecure\",\"level\":\"info\",\"msg\":\"\",\"request_User-Agent\":\"Dart/3.5 (dart:io)\",\"time\":\"2025-03-04T18:05:02-07:00\"}"
crowdsec | time="2025-03-04T18:05:02-07:00" level=warning msg="failed to run filter : invalid character '7' after top-level value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=lingering-sun name=child-crowdsecurity/traefik-logs stage=s01-parse
crowdsec | time="2025-03-05T13:25:50-07:00" level=error msg="UnmarshalJSON : unexpected end of JSON input" line="{\"ClientAddr\":\"86.173.7.100:62653\",\"ClientHost\":\"86.173.7.100\",\"ClientPort\":\"62653\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":16316,\"DownstreamStatus\":200,\"Duration\":4863609,\"OriginContentSize\":16316,\"OriginDuration\":3957836,\"OriginStatus\":200,\"Overhead\":905773,\"RequestAddr\":\"plex.example.com\",\"RequestContentSize\":0,\"RequestCount\":63334,\"RequestHost\":\"plex.example.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/web/chunk-5225-ff4db6bba00dcffe048c-plex-4.143.0-30c4cf2.css\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/2.0\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"id-media-plex-rtr@file\",\"ServiceAddr\":\"192.168.1.95:32400\",\"ServiceName\":\"id-media-plex-srv@file\",\"ServiceURL\":\"http://192.168.1.95:32400"
crowdsec | time="2025-03-05T13:25:50-07:00" level=warning msg="failed to run filter : unexpected end of JSON input (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=lingering-sun name=child-crowdsecurity/traefik-logs stage=s01-parse
crowdsec | time="2025-03-05T17:34:13-07:00" level=error msg="UnmarshalJSON : unexpected end of JSON input" line="{\"ClientAddr\":\"192.168.1.105:55224\",\"ClientHost\":\"192.168.1.105\",\"ClientPort\":\"55224\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":21,\"DownstreamStatus\":499,\"Duration\":666895213,\"OriginContentSize\":21,\"OriginDuration\":663485804,\"OriginStatus\":499,\"Overhead\":3409409,\"RequestAddr\":\"immich.example.com\",\"RequestContentSize\":0,\"RequestCount\":74003,\"RequestHost\":\"immich.example.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/assets/d87f4849-e5fc-41d6-80ae-8b9dc4b0ebb6/thumbnail?size=thumbnail\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"id-services-immich-rtr@file\",\"ServiceAddr\":\"192.168.1.93:2283\",\"ServiceName\":\"id-services-immich-srv@file\",\"ServiceURL\":\"http://192.168.1.93:2283\",\"SpanId\":\"0000000000000000\",\"StartLocal\":\"2025-03-05T17:34:12.309937454-07:00\",\"StartUTC\":\"2025-03-06T00:34:12.309937454Z\",\"TLSCipher\":\"TLS_CHACHA20_POLY1305_SHA256\",\"TLSVersion\":\"1.3\",\"TraceId\":\"00000000000000000000000000000000\",\"entryPointName\":\"websecure\",\"level\":\"info\",\"msg\":\"\",\"request_User-Agent\":\"Da"
Log lines 1, 2, & 3 relates to the following access.log entry for Traefik:
{"ClientAddr":"192.168.1.105:56612","ClientHost":"192.168.1.105","ClientPort":"56612","ClientUsername":"-","DownstreamContentSize":9836,"DownstreamStatus":200,"Duration":557951410,"OriginContentSize":9836,"OriginDuration":557207858,"OriginStatus":200,"Overhead":743552,"RequestAddr":"immich.example.com","RequestContentSize":0,"RequestCount":16027,"RequestHost":"immich.example.com","RequestMethod":"GET","RequestPath":"/api/assets/15accaea-6c75-4168-a470-9af475f9a8d3/thumbnail?size=thumbnail","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"RouterName":"id-services-immich-rtr@file","ServiceAddr":"192.168.1.93:2283","ServiceName":"id-services-immich-srv@file","ServiceURL":"http://192.168.1.93:2283","SpanId":"0000000000000000","StartLocal":"2025-03-04T18:05:02.356967516-07:00","StartUTC":"2025-03-05T01:05:02.356967516Z","TLSCipher":"TLS_CHACHA20_POLY1305_SHA256","TLSVersion":"1.3","TraceId":"00000000000000000000000000000000","entryPointName":"websecure","level":"info","msg":"","request_User-Agent":"Dart/3.5 (dart:io)","time":"2025-03-04T18:05:02-07:00"}
Log line 5, 6, & 7 relate to the following access.log entry for Traefik:
{"ClientAddr":"192.168.1.93:48714","ClientHost":"192.168.1.93","ClientPort":"48714","ClientUsername":"-","DownstreamContentSize":1752,"DownstreamStatus":200,"Duration":3285004,"OriginContentSize":1752,"OriginDuration":2669840,"OriginStatus":200,"Overhead":615164,"RequestAddr":"prometheus.example.com","RequestContentSize":0,"RequestCount":74449,"RequestHost":"prometheus.example.com","RequestMethod":"GET","RequestPath":"/query","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"RouterName":"id-services-prometheus-rtr@file","ServiceAddr":"192.168.1.93:9090","ServiceName":"id-services-prometheus-srv@file","ServiceURL":"http://192.168.1.93:9090","SpanId":"0000000000000000","StartLocal":"2025-03-05T17:45:04.017493642-07:00","StartUTC":"2025-03-06T00:45:04.017493642Z","TLSCipher":"TLS_CHACHA20_POLY1305_SHA256","TLSVersion":"1.3","TraceId":"00000000000000000000000000000000","entryPointName":"websecure","level":"info","msg":"","request_User-Agent":"Uptime-Kuma/1.23.16","time":"2025-03-05T17:45:04-07:00"}
To Reproduce Add the following collections to crowdsec:
- crowdsecurity/traefik
Add the following entries to acquis.yaml:
source: file
filenames:
- /logs/traefik/*.log
log_level: warn
labels:
type: traefik
Expected behavior The traefik access.log to be parsed without issue
Screenshots n/a
Additional context Relevant traefik.yaml content:
log:
filePath: /srv/logs/traefik.log
level: WARN
accessLog:
filePath: /srv/logs/access.log
addInternals: true
format: json
filters:
statusCodes:
- 200-299
- 400-499
- 500-599
bufferingSize: 0
fields:
headers:
defaultMode: drop
names:
User-Agent: keep
We are ensuring traefik logs in json mode.
I had the same problem. It seems that crowdsec is truncating the beginning or end of random lines when reading traefik-access.log, so the parser doesn't recognize it as json format.
I dug into the logs and the lines that crowdsec considered to be incorrect are correct in the traefik-access.log file.
Crowdsec log:
time="2025-05-31T18:35:05+02:00" level=error msg="UnmarshalJSON : unexpected end of JSON input" line="{\"ClientAddr\":\"192.168.103.111:44924\",\"ClientHost\":\"192.168.103.111\",\"ClientPort\":\"44924\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":8193,\"DownstreamStatus\":200,\"Duration\":3483900,\"OriginContentSize\":8193,\"OriginDuration\":3280472,\"OriginStatus\":200,\"Overhead\":203428,\"RequestAddr\":\"jellyfin.mydomain.tld\",\"RequestContentSize\":0,\"RequestCount\":74199,\"RequestHost\":\"jellyfin.mydomain.tld\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/Shows/ce1d66b953ab7020898f467fce3d40d5/Seasons\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestSchem"
Traefik access log:
{"ClientAddr":"192.168.103.111:44924","ClientHost":"192.168.103.111","ClientPort":"44924","ClientUsername":"-","DownstreamContentSize":8193,"DownstreamStatus":200,"Duration":3483900,"OriginContentSize":8193,"OriginDuration":3280472,"OriginStatus":200,"Overhead":203428,"RequestAddr":"jellyfin.mydomain.tld","RequestContentSize":0,"RequestCount":74199,"RequestHost":"jellyfin.mydomain.tld","RequestMethod":"GET","RequestPath":"/Shows/ce1d66b953ab7020898f467fce3d40d5/Seasons","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"RouterName":"jellyfin@file","ServiceAddr":"jellyfin.mydomain.tld","ServiceName":"jellyfin@file","ServiceURL":"https://jellyfin.mydomain.tld","StartLocal":"2025-05-31T18:35:04.996066781+02:00","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","request_User-Agent":"node","time":"2025-05-31T18:35:04+02:00"}
Just discovered the same logs on my server.
time="2025-08-02T19:12:53+10:00" level=warning msg="failed to run filter : unexpected end of JSON input (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=falling-water name=child-crowdsecurity/traefik-logs stage=s01-parse
time="2025-08-02T19:12:53+10:00" level=error msg="UnmarshalJSON : invalid character 'u' looking for beginning of value" line="uestMethod\":\"POST\",\"RequestPath\":\"/plugins/unassigned.devices.preclear/include/Preclear.php\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/2.0\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"apollo@file\",\"ServiceAddr\":\"10.0.0.100:8088\",\"ServiceName\":\"apollo@file\",\"ServiceURL\":\"http://10.0.0.100:8088/\",\"StartLocal\":\"2025-08-02T19:12:53.673292055+10:00\",\"TLSCipher\":\"TLS_AES_128_GCM_SHA256\",\"TLSVersion\":\"1.3\",\"entryPointName\":\"https\",\"level\":\"info\",\"msg\":\"\",\"time\":\"2025-08-02T19:12:53+10:00\"}"
time="2025-08-02T19:12:53+10:00" level=warning msg="failed to run filter : invalid character 'u' looking for beginning of value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=falling-water name=child-crowdsecurity/traefik-logs stage=s01-parse
time="2025-08-02T19:13:05+10:00" level=error msg="UnmarshalJSON : unexpected end of JSON input" line="{\"ClientAddr\":\"10.0.0.100:54604\",\"ClientHost\":\"10.0.0.100\",\"ClientPort\":\"54604\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":68,\"DownstreamStatus\":200,\"Duration\":2513654,\"Origi"
time="2025-08-02T19:13:05+10:00" level=warning msg="failed to run filter : unexpected end of JSON input (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=falling-water name=child-crowdsecurity/traefik-logs stage=s01-parse
time="2025-08-02T19:13:05+10:00" level=error msg="UnmarshalJSON : invalid character 'C' in literal null (expecting 'u')" line="nContentSize\":68,\"OriginDuration\":2410167,\"OriginStatus\":200,\"Overhead\":103487,\"RequestAddr\":\"example.com\",\"RequestContentSize\":0,\"RequestCount\":750,\"RequestHost\":\"example.com\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/server/version-check\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RequestScheme\":\"https\",\"RetryAttempts\":0,\"RouterName\":\"immich-internal@docker\",\"ServiceAddr\":\"172.18.2.5:2283\",\"ServiceName\":\"immich-internal@docker\",\"ServiceURL\":\"http://172.18.2.5:2283\",\"StartLocal\":\"2025-08-02T19:13:05.045561358+10:00\",\"TLSCipher\":\"TLS_AES_128_GCM_SHA256\",\"TLSVersion\":\"1.3\",\"entryPointName\":\"https\",\"level\":\"info\",\"msg\":\"\",\"time\":\"2025-08-02T19:13:05+10:00\"}"
How do we know if this is (or isn't) a concern? My concern is that my bouncer is unable to filter some logs and block access as the logs can't be read properly.
Is there a fix for this?
Here is relevant access long configuration from traefik:
accesslog:
filePath: "/var/log/traefik/logs/access.log"
format: json
filters:
statusCodes:
- "200-299"
- "400-599"
bufferingSize: 0
fields:
headers:
defaultMode: drop
names:
StartUTC: drop
User-Agent: keep
