tlsClientAuth for Appsec isn't working

[srkoster]

#222 made some changes to support TLS for appsec, however they aren't working.

The main reason is that the PR creates the appsec-certificate.yaml for the Appsec deployment and passes it to the CLIENT_CERT_FILE of the container. However, the certificate isn't mounted and passed to the LAPI container. Therefore the Appsec containers can't be authenticated, fail to start and log the following error time="level=fatal msg="unable to start crowdsec routines: authenticate watcher (): API error: client certificate OU [appsec-ou] doesn't match expected OU [agent-ou]"

The agent certificate is already passed to the CLIENT_CERT_FILE of the LAPI container and I don't think it's possible to pass two certificates for agent authentication to the LAPI.

[github-actions[bot]]

@srkoster: Thanks for opening an issue, it is currently awaiting triage.

If you haven't already, please provide the following information:

• kind : bug, enhancementor documentation
• area : agent, appsec, configuration, cscli, local-api

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the forked project rr404/oss-governance-bot repository.

[github-actions[bot]]

@srkoster: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind bug
• /kind documentation
• /kind enhancement

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the forked project rr404/oss-governance-bot repository.

[srkoster]

/kind bug /area appsec

[mmetc]

Hello @srkoster

The agent certificate is passed to the lapi because it's needed by cscli to connect.

Actually what is missing in your case is the configuration for LAPI to accept both client certificates, in crowdsec-values.yaml:

lapi:
  env:
    - name: AGENTS_ALLOWED_OU
      value: agent-ou,appsec-ou

I don't think we want them as default because they depend on how the certificates are created.

[srkoster]

@mmetc thanks for the settings, I'll try them and let you know

[srkoster]

I can confirm that added the settings mentioned in https://github.com/crowdsecurity/helm-charts/issues/239#issuecomment-2643219817 solve the issue.

However, I still can't get my Traefik Crowdsec bouncer to work with Appsec and TLS. I've set the bouncer to use https and get this error:

DEBUG: CrowdsecBouncerTraefikPlugin: 2025/02/08 13:05:29 handleNextServeHTTP ip:192.168.1.13 isWaf:true appsecQuery:unreachable Get "https://crowdsec-appsec-service.crowdsec:7422/": http: server gave HTTP response to HTTPS client

Setting the bouncer to use http prevents it from registering at the Lapi since that is using https and will give the following errors:

2025/02/08 13:11:57 http: TLS handshake error from 10.114.6.151:39464: remote error: tls: bad certificate
time="2025-02-08T13:11:57Z" level=error msg="Unauthorized request from '10.114.2.83:59008' (real IP = 192.168.1.13)" type=appsec
time="2025-02-08T13:11:57Z" level=error msg="Error performing request: Head \"https://crowdsec-service.crowdsec:8080/v1/decisions/stream\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

@mmetc can you confirm that Appsec is only using TLS to connect with the Lapi, but not to set the scheme to https for

- containerPort: 7422
   name: appsec
   protocol: TCP

If that's the case, then either the Appsec component has to be adapted to server https when TLS is active or the bouncer needs to support different schemes for Appsec and Lapi as requested in https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/issues/131

[mathieuHa]

I can confirm that added the settings mentioned in #239 (comment) solve the issue.

However, I still can't get my Traefik Crowdsec bouncer to work with Appsec and TLS. I've set the bouncer to use https and get this error:

DEBUG: CrowdsecBouncerTraefikPlugin: 2025/02/08 13:05:29 handleNextServeHTTP ip:192.168.1.13 isWaf:true appsecQuery:unreachable Get "https://crowdsec-appsec-service.crowdsec:7422/": http: server gave HTTP response to HTTPS client

Setting the bouncer to use http prevents it from registering at the Lapi since that is using https and will give the following errors:

2025/02/08 13:11:57 http: TLS handshake error from 10.114.6.151:39464: remote error: tls: bad certificate
time="2025-02-08T13:11:57Z" level=error msg="Unauthorized request from '10.114.2.83:59008' (real IP = 192.168.1.13)" type=appsec
time="2025-02-08T13:11:57Z" level=error msg="Error performing request: Head \"https://crowdsec-service.crowdsec:8080/v1/decisions/stream\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

@mmetc can you confirm that Appsec is only using TLS to connect with the Lapi, but not to set the scheme to https for

- containerPort: 7422
   name: appsec
   protocol: TCP

If that's the case, then either the Appsec component has to be adapted to server https when TLS is active or the bouncer needs to support different schemes for Appsec and Lapi as requested in maxlerebourg/crowdsec-bouncer-traefik-plugin#131

Hi @srkoster,

My Crowdsec plugin broke also after helm upgrade.
I will look in the following days how to handle this with the https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/ plugin
In the meantime I have reverted

[blotus]

Hello,

Sorry for the confusion, but at the moment TLS authentication between a bouncer and the appsec datasource is not supported.

This is quite tricky to implement: by default, we rely on the bouncer sending the API key to the log processor, which in turn will make a "fake" request to LAPI using this API key to validate it.

If the bouncer is using TLS auth, we cannot get an API key from the request, so we cannot check its validity.

We are exploring how to add support for it in crowdsec (the most straightforward solution would likely be to add a new option to the appsec datasource to give it a CA to validate the client certificates with), but it's something that cannot be fixed purely in helm at the moment.