Problem with Caddy parser when Caddy is behind Cloudflare Proxy

[jmarsh88]

Hi,

It looks like the Caddy log parser does not read the X-Forwarded-For headers to obtain source/client IP addresses. This is a problem if you're running Caddy behind a proxy such as Cloudflare, as the IP that is being parsed belongs to Cloudflare and not the client.

I have set use_forwarded_for_headers in Config.yaml to true as per https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/ but that's not made any difference.

Hoping that a minor change to the caddy-logs.yaml file is all that's needed to fix this? (https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/caddy-logs.yaml)

Caddy (v2.6.2) is configured with a trusted proxy list, containing all Cloudflare IP ranges and outputting logs in Caddy's default JSON format.

Log Example: line: {"level":"info","ts":1673579550.8916137,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"162.158.2.69","remote_port":"29702","proto":"HTTP/2.0","method":"GET","host":"my.domain.name","uri":"/web/33b04ba592abc68c06af.woff2","headers":{"If-Modified-Since":["Tue, 29 Nov 2022 18:42:54 GMT"],"Referer":["https://my.domain.name/web/serviceworker.js"],"Priority":["u=1"],"Cf-Connecting-Ip":["2001:8004:1102:af7b:5de8:787e:a8a2:4288"],"Cf-Ipcountry":["AU"],"Accept-Encoding":["gzip"],"Cf-Ray":["788af360ec4f5a73-MEL"],"Dnt":["1"],"Sec-Fetch-Dest":["empty"],"If-None-Match":[""1d904226429b9fc""],"Sec-Gpc":["1"],"Accept-Language":["en-AU,en"],"Sec-Fetch-Mode":["cors"],"Accept":["/"],"Sec-Fetch-Site":["same-origin"],"Cf-Pseudo-Ipv4":["255.184.204.165"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["2001:8004:1102:af7b:5de8:787e:a8a2:4288"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{"scheme":"https"}"],"User-Agent":["Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Mobile Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"my.domain.name","client_common_name":"","client_serial":"423664799579409626483383746572328861702120547"}},"user_id":"","duration":0.000680323,"size":0,"status":304,"resp_headers":{"Server":["Caddy","Kestrel"],"Alt-Svc":["h3=":443"; ma=2592000"],"Etag":[""1d904226429b9fc""],"Last-Modified":["Tue, 29 Nov 2022 18:42:54 GMT"],"X-Response-Time-Ms":["0"],"Content-Type":["font/woff2"],"Date":["Fri, 13 Jan 2023 03:12:30 GMT"],"Accept-Ranges":["bytes"]}}

[LaurenceJJones]

I have set use_forwarded_for_headers in Config.yaml to true as per https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/ but that's not made any difference.

That is just for crowdsec http api not for crowdsec to know to change the log patterns.

Hoping that a minor change to the caddy-logs.yaml file is all that's needed to fix this? (https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/caddy-logs.yaml)

We could but then we open a whole load of risks to people who dont use cloudflare with caddy as crowdsec itself would need to know that the IP is trusted but that is within caddy config not ours.

Let me think on this, but im sure in your caddy configuration you can set the remote IP to be the CF-Connecting-IP, however, im not a user of caddy and are not talking from experience.

[jmarsh88]

I have set use_forwarded_for_headers in Config.yaml to true as per https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/ but that's not made any difference.

That is just for crowdsec http api not for crowdsec to know to change the log patterns.

That makes sense.

Hoping that a minor change to the caddy-logs.yaml file is all that's needed to fix this? (https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/caddy-logs.yaml)

We could but then we open a whole load of risks to people who dont use cloudflare with caddy as crowdsec itself would need to know that the IP is trusted but that is within caddy config not ours.

Rather than change it for everyone, could it be something that can be enabled/set on a particlar Crowdsec install if/when required?

Let me think on this, but im sure in your caddy configuration you can set the remote IP to be the CF-Connecting-IP, however, im not a user of caddy and are not talking from experience.

Thanks, appreciate it. I don't think this is possible just yet(see https://github.com/caddyserver/caddy/issues/4924 & https://github.com/caddyserver/caddy/pull/5104). Most of the applications I use already allow you to set the client IP header to use, such as X-Forwarded-For or CF-Connecting-IP, so it would be great if the parser could support this too :)

[laurentftech]

Hello. Newbie here, I am just trying to make Crowdsec working with Caddy with Cloudflared. Is there any progress on this issue? Thank you.

[LaurenceJJones]

So now since it is implemented I tried to update the caddy parser #806 , however, I am running into a CI issue. The parser works but I cannot merge as it will break the hub CI tests

[ryanwinter]

Hi @LaurenceJJones,

I was looking at the referenced PR, but it didnt seem to be related to this issue. Maybe I was looking at the wrong thing :)

Good to know about the use_forwarded_for_header not being something related to parsers using a different IP source. I think that maybe I can rewrite the IP in at the caddy side.

[LaurenceJJones]

I was looking at the referenced PR, but it didnt seem to be related to this issue. Maybe I was looking at the wrong thing :)

It was related before caddy supported setting a trusted IP and trusted headers, there was a IP within the log line on a different key than client_ip the PR changed it so we now use client_ip as this is the IP or the trusted header IP

[ryanwinter]

It was related before caddy supported setting a trusted IP and trusted headers, there was a IP within the log line on a different key than client_ip the PR changed it so we now use client_ip as this is the IP or the trusted header IP

Figured it out :) I think being a more recent feature, I found it a bit confusing :)