falco
falco copied to clipboard
falcosecurity
Default helm chart install fails to verify certificate
Describe the bug
Deployment of helm chart with default values file results in falco init container failing with:
falco-5949c9ccbc-vdwk6 falcoctl-artifact-install {"level":"ERROR","msg":"error while verifying signature for ghcr.io/falcosecurity/rules/falco-rules@sha256:d4c03e000273a0168ee3d9b3dfb2174e667b93c9bfedf399b298ed70f37d623b: getting Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 8.root.json: Get \"https://tuf-repo-cdn.sigstore.dev/8.root.json\": tls: failed to verify certificate: x509: certificate is valid for 22fa645d946ce89e04f0a8de95f25eed.3657b8f779991c6666eb76c90b587c3c.traefik.default, not tuf-repo-cdn.sigstore.dev\nremote status:{\n\t\"mirror\": \"https://tuf-repo-cdn.sigstore.dev\",\n\t\"metadata\": {}\n}","timestamp":"2024-02-27 10:11:08"}
How to reproduce it
Install helm chart using provided values.yaml.
Expected behaviour
Falco starts and starts collecting data.
Screenshots N/A
Environment
-
Falco version: 0.37.1
-
Cloud provider or hardware configuration: k3s cluster running 3 Ubuntu Jammy nodes.
-
OS: PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy
-
Kernel: Linux core-k8s 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:26:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
-
Installation method: Helm
Additional context N/A
I have just tried to install Falco and it appears to be working. I thought this was a transient issue with Sigstore but maybe something is off with your setup?
{"level":"INFO","msg":"Resolving dependencies ...","timestamp":"2024-02-27 10:45:19"}
{"level":"INFO","msg":"Installing artifacts","refs":["ghcr.io/falcosecurity/rules/falco-rules:3"],"timestamp":"2024-02-27 10:45:19"}
{"level":"INFO","msg":"Preparing to pull artifact","ref":"ghcr.io/falcosecurity/rules/falco-rules:3","timestamp":"2024-02-27 10:45:19"}
{"level":"INFO","msg":"Pulling layer 1e72f9c4d8fe","timestamp":"2024-02-27 10:45:20"}
{"level":"INFO","msg":"Pulling layer 2e91799fee49","timestamp":"2024-02-27 10:45:20"}
{"level":"INFO","msg":"Pulling layer d4c03e000273","timestamp":"2024-02-27 10:45:20"}
{"digest":"ghcr.io/falcosecurity/rules/falco-rules@sha256:d4c03e000273a0168ee3d9b3dfb2174e667b93c9bfedf399b298ed70f37d623b","level":"INFO","msg":"Verifying signature for artifact","timestamp":"2024-02-27 10:45:20"}
{"level":"INFO","msg":"Signature successfully verified!","timestamp":"2024-02-27 10:45:21"}
{"file":"falco_rules.yaml.tar.gz","level":"INFO","msg":"Extracting and installing artifact","timestamp":"2024-02-27 10:45:21","type":"rulesfile"}
{"digest":"sha256:d4c03e000273a0168ee3d9b3dfb2174e667b93c9bfedf399b298ed70f37d623b","directory":"/rulesfiles","level":"INFO","msg":"Artifact successfully installed","name":"ghcr.io/falcosecurity/rules/falco-rules:3","timestamp":"2024-02-27 10:45:21","type":"rulesfile"}
I see that in your logs it says
x509: certificate is valid for 22fa645d946ce89e04f0a8de95f25eed.3657b8f779991c6666eb76c90b587c3c.traefik.default, not tuf-repo-cdn.sigstore.dev
Could it be that traefik is intercepting this communication? I am not familiar with traefik, does it need additional configuration? Without it I see a certificate that looks valid:
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1D4
verify return:1
depth=0 CN = tuf-repo-cdn.sigstore.dev
verify return:1
---
Certificate chain
0 s:CN = tuf-repo-cdn.sigstore.dev
i:C = US, O = Google Trust Services LLC, CN = GTS CA 1D4
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Feb 1 14:27:23 2024 GMT; NotAfter: May 1 15:21:57 2024 GMT
1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1D4
i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 13 00:00:42 2020 GMT; NotAfter: Sep 30 00:00:42 2027 GMT
2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Jun 19 00:00:42 2020 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
Server certificate
-----BEGIN CERTIFICATE-----
MIIFdjCCBF6gAwIBAgIRAL2KVpEjIp2lCpHuW1r4c4cwDQYJKoZIhvcNAQELBQAw
RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
TEMxEzARBgNVBAMTCkdUUyBDQSAxRDQwHhcNMjQwMjAxMTQyNzIzWhcNMjQwNTAx
MTUyMTU3WjAkMSIwIAYDVQQDExl0dWYtcmVwby1jZG4uc2lnc3RvcmUuZGV2MIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAopzg41O/PDBck2WMmRk5PCTI
kzK8Q4hzTpV8jpkJ2iQ/ohlXGYOLxx8fU5HS/FlxrV6V1uOIGud6OMAJdw2ly70J
I1qOfJQahev6e6WSLQEu9QBMnsQhB+EsrCQ7hFF8PXRYJ+Z/IH4UFtjGtGVFpNJc
dtlivy3eNoPogNFquF2728vh932DzA4RZVKXg6Sv4ZaGPHjImG38OAyctmmoYDS0
HhCYIRu4wCXPQMmPobrxE0MPwZIDcOY6Umw02i0zKR9Dh7zN0Fm3ad99OQKxLhs5
chqn/wf+xgkReZpNhb0lLexSfKKS011aO9BmLyfengLMvSDGdckB6IRq8LwsTwID
AQABo4ICfzCCAnswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMB
MAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFL7xRRChJRCJ7Sx6tNBrJBqnCnZGMB8G
A1UdIwQYMBaAFCXiGA6yV5GUKuXUXYaQg95Ts7iSMHgGCCsGAQUFBwEBBGwwajA1
BggrBgEFBQcwAYYpaHR0cDovL29jc3AucGtpLmdvb2cvcy9ndHMxZDQvZUN4T0xo
MEZ0VVkwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0
czFkNC5kZXIwJAYDVR0RBB0wG4IZdHVmLXJlcG8tY2RuLnNpZ3N0b3JlLmRldjAh
BgNVHSAEGjAYMAgGBmeBDAECATAMBgorBgEEAdZ5AgUDMDwGA1UdHwQ1MDMwMaAv
oC2GK2h0dHA6Ly9jcmxzLnBraS5nb29nL2d0czFkNC85Q0liU0l6MU1Uay5jcmww
ggEDBgorBgEEAdZ5AgQCBIH0BIHxAO8AdQB2/4g/Crb7lVHCYcz1h7o0tKTNuync
aEIKn+ZnTFo6dAAAAY1lSCZJAAAEAwBGMEQCIAK6S6lBc1JOlRgWpxXWefPtLD2O
gkA3326Jv/1qnn1HAiByAWJDL4ywaLvPMvfOwhpy3nlxGRMTFobY0i7ppVvjagB2
AEiw42vapkc0D+VqAvqdMOscUgHLVt0sgdm7v6s52IRzAAABjWVIJjoAAAQDAEcw
RQIhAIJSzahBv4wVftchuRxd2wYwnYZ/pIcPquKZjCoSVgHeAiBaJ+fUciE4Y818
TkSnHvJJe/Ksg8r7D3CV6Y6TsBuiczANBgkqhkiG9w0BAQsFAAOCAQEADos/fRBM
0ci0hA18BLOMimhVPgJ4K80CG7sA6AJ6KufCt425tUye4LYCVORn71wPCy+U8n9r
dDedaDJnaJLI0rLtDZ4GTcolQEJnYe5KzR5XOvT7Xa31gHHrClMaYa2S/xGQ7ujL
J/H3Ws+JvJvviwp17BzNC7MmZFONhtB7m91Z0uwBBPezinZb8aN57JwQiskqqhzV
TNKdBtI9k6YjjwCDtkegM8An8292DTTQ6n2nyH0oEMHPNkF3Ppxyt+/CwLPEgNzZ
uqXQ4BvRCNjXv/lLsXGofv/W5TQ3bvSxC1/HVGIN0uHBalS/3CPnDhxtspTCSMTv
UJ7RkNwrd5yXhA==
-----END CERTIFICATE-----
subject=CN = tuf-repo-cdn.sigstore.dev
issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1D4
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4708 bytes and written 407 bytes
Verification: OK
---
Yeah, sigstore works and I'm able to curl file from any node directly, it just won't work from within kubernetes. If it was something to do with traefik, I think it shouldn't work from any of the nodes, not just worker ones which all are running on the same subnet, in same DC.
I am not familiar with Traefik configuration but I don't think Falco on its own could generate this error. You can try to curl or run openssl s_client -connect tuf-repo-cdn.sigstore.dev:443 -prexit from working and not working pods to see if there is any difference. If you find a solution please let us know so we know what to recommend if someone comes with similar problems.
If all else fails, or if there is no way to serve the sigstore certificate in those pods, there is an environment variable FALCOCTL_ARTIFACT_NOVERIFY that you can set on the falcoctl pods to skip signature verification.
FALCOCTL_ARTIFACT_NOVERIFY worked, thanks!
The issue we had is that Traefik on-prem deployment is impacted by some weird network configuration.
Falco works as expected. Thanks for helping <3