falco icon indicating copy to clipboard operation
falco copied to clipboard

Unable to get manifest: unable to fetch reference || tls: failed to verify certificate: x509: certificate signed by unknown authority

Open Su-HackZ opened this issue 5 months ago • 16 comments

Describe the bug

We have private k8s cluster and we use proxy to communicate for outside the N/W. After adding the proxy under ENV for http_proxy & https_proxy in values.yaml

we are getting below error

{"level":"INFO","msg":"Resolving dependencies ...","timestamp":"2025-06-09 06:24:32"} {"level":"ERROR","msg":"unable to get manifest: unable to fetch reference "ghcr.io/falcosecurity/rules/falco-rules:3": Get "https://ghcr.io/v2/falcosecurity/rules/falco-rules/manifests/3": tls: failed to verify certificate: x509: certificate signed by unknown authority","timestamp":"2025-06-09 06:24:32"} {"level":"INFO","msg":"Resolving dependencies ...","timestamp":"2025-06-09 06:25:01"}

How to reproduce it

Add under extra and and falcoctl for falco Helm

extra: #--Extra environment variables that will be pass onto Falco containers. env: - name: https_proxy value: "http://myProxy.com:8080" - name: http_proxy value: "http://myProxy.com:8080"
falcoctl: image: # -- The image pull policy. pullPolicy: IfNotPresent # -- The image registry to pull from. registry: PrivateArtifactory.com # -- The image repository to pull from. repository: docker/falcosecurity/falcoctl # -- The image tag to pull. tag: "0.11.0" artifact: # -- Runs "falcoctl artifact install" command as an init container. It is used to install artfacts before # Falco starts. It provides them to Falco by using an emptyDir volume. install: enabled: true #--Extra environment variables that will be pass onto falcoctl-artifact-install init container. env: - name: https_proxy value: "http://myProxy.com:8080" - name: http_proxy value: "http://myProxy.com:8080"

Expected behaviour

{"level":"INFO","msg":"Resolving dependencies ...","timestamp":"2025-06-09 06:24:32"} {"level":"ERROR","msg":"unable to get manifest: unable to fetch reference "ghcr.io/falcosecurity/rules/falco-rules:3": Get "https://ghcr.io/v2/falcosecurity/rules/falco-rules/manifests/3": tls: failed to verify certificate: x509: certificate signed by unknown authority","timestamp":"2025-06-09 06:24:32"} {"level":"INFO","msg":"Resolving dependencies ...","timestamp":"2025-06-09 06:25:01"}

Environment

  • Falco version: latest Helm
  • System info: K8s (AKS)
  • Cloud provider or hardware configuration:
  • OS: N/A
  • Kernel: N/A
  • Installation method: HELM

Su-HackZ avatar Jun 09 '25 05:06 Su-HackZ

without proxy variable, it is not been able to fetch index

{"level":"ERROR","msg":"unable to fetch index "falcosecurity" with URL "https://falcosecurity.github.io/falcoctl/index.yaml": unable to fetch index: cannot fetch index: Get "https://falcosecurity.github.io/falcoctl/index.yaml": dial tcp 185.199.109.153:443: i/o timeout","timestamp":"2025-06-09 06:00:30"}


with proxy

{"level":"INFO","msg":"Resolving dependencies ...","timestamp":"2025-06-09 06:24:32"} {"level":"ERROR","msg":"unable to get manifest: unable to fetch reference "ghcr.io/falcosecurity/rules/falco-rules:3": Get "[https://ghcr.io/v2/falcosecurity/rules/falco-rules/manifests/3](https://ghcr.io/v2/falcosecurity/rules/falco-rules/manifests/3%5C)": tls: failed to verify certificate: x509: certificate signed by unknown authority","timestamp":"2025-06-09 06:24:32"} {"level":"INFO","msg":"Resolving dependencies ...","timestamp":"2025-06-09 06:25:01"}

Su-HackZ avatar Jun 09 '25 06:06 Su-HackZ

Hey! I found this one that can be relevant: https://github.com/falcosecurity/falco/issues/3119

FedeDP avatar Jun 10 '25 09:06 FedeDP

Hey! I found this one that can be relevant: #3119

Getting same with FALCOCTL_ARTIFACT_NOVERIFY = true

falcoctl: artifact: install: enabled: true env: - name: https_proxy value: "http://myproxy.com:8080/" - name: http_proxy value: "http://myproxy.com:8080/" - name: FALCOCTL_ARTIFACT_NOVERIFY value: "true"

Su-HackZ avatar Jun 10 '25 10:06 Su-HackZ

Is it still saying that

tls: failed to verify certificate: x509: certificate signed by unknown authority

?

FedeDP avatar Jun 10 '25 15:06 FedeDP

ll saying that

Yes

{"level":"ERROR","msg":"unable to fetch index "falcosecurity" with URL "https://falcosecurity.github.io/falcoctl/index.yaml": unable to fetch index: cannot fetch index: Get "https://falcosecurity.github.io/falcoctl/index.yaml": tls: failed to verify certificate: x509: certificate signed by unknown authority","timestamp":"2025-06-11 05:39:19"}

Su-HackZ avatar Jun 11 '25 05:06 Su-HackZ

As far as i see, it is failing here: https://github.com/falcosecurity/falcoctl/blob/6d9b57671f4c78a66af659288c0fb45fdac7d503/pkg/index/fetch/http/fetcher.go#L37

Since here we are just using net/http golang package, i assume there must be some misconfiguration in the proxy; have you tried to use uppercase for proxy env variables, eg:

  • HTTP_PROXY
  • HTTPS_PROXY (found this one too https://github.com/falcosecurity/falcoctl/issues/290)

FedeDP avatar Jun 11 '25 07:06 FedeDP

As far as i see, it is failing here: https://github.com/falcosecurity/falcoctl/blob/6d9b57671f4c78a66af659288c0fb45fdac7d503/pkg/index/fetch/http/fetcher.go#L37

Since here we are just using net/http golang package, i assume there must be some misconfiguration in the proxy; have you tried to use uppercase for proxy env variables, eg:

Yes, Already tested with uppercase too

Su-HackZ avatar Jun 11 '25 08:06 Su-HackZ

@Su-HackZ Can you check if http://myproxy.com:8080/ (or whatever URL you used) actually resolves to your proxy from within the falcoctl's container?

leogr avatar Jun 12 '25 07:06 leogr

Maybe is just a mistyping issue, but I see you wrote:

  • name: https_proxy value: "http://myproxy.com:8080/"

Could it be:

  • name: https_proxy value: "https://myproxy.com:8080/"

?

ekoops avatar Jun 12 '25 07:06 ekoops

@Su-HackZ Can you check if http://myproxy.com:8080/ (or whatever URL you used) actually resolves to your proxy from within the falcoctl's container?

Checked with http://myproxy.com:8080/ getting same

env:

  • name: https_proxy value: "http://myproxy.com:8080/"
  • name: http_proxy value: "http://myproxy.com:8080/"
  • name: FALCOCTL_ARTIFACT_NOVERIFY value: "true"

Su-HackZ avatar Jun 12 '25 09:06 Su-HackZ

Maybe is just a mistyping issue, but I see you wrote:

  • name: https_proxy value: "http://myproxy.com:8080/"

Could it be:

  • name: https_proxy value: "https://myproxy.com:8080/"

?

It is not typo, for https_proxy we are using same value as http_proxy http://myproxy.com:8080

Su-HackZ avatar Jun 12 '25 09:06 Su-HackZ

Checked with http://myproxy.com:8080/ getting same

If you meant that the proxy isn't resolving or responding as expected, the issue isn't in falcoctl.

For example, if you run from within the falcoctl container:

curl -v --proxy https://myproxy.com:8080 https://example.com/

and you get the same problem, likely the root cause is in the way the container reaches the proxy (eg. bad DNS?)

leogr avatar Jun 12 '25 10:06 leogr

Checked with http://myproxy.com:8080/ getting same

If you meant that the proxy isn't resolving or responding as expected, the issue isn't in falcoctl.

For example, if you run from within the falcoctl container:

curl -v --proxy https://myproxy.com:8080 https://example.com/ and you get the same problem, likely the root cause is in the way the container reaches the proxy (eg. bad DNS?)

The DNS worked fine , falcoctl does not have shell the status is waiting, I did curl from another pod though, Having http_proxy as ENV variable and from the pod it was able to curl the falcoctl index

This is very strange behavior, we can see with openssl command that there is no issue with proxy too....

Su-HackZ avatar Jun 12 '25 12:06 Su-HackZ

Checked with http://myproxy.com:8080/ getting same

If you meant that the proxy isn't resolving or responding as expected, the issue isn't in falcoctl. For example, if you run from within the falcoctl container: curl -v --proxy https://myproxy.com:8080 https://example.com/ and you get the same problem, likely the root cause is in the way the container reaches the proxy (eg. bad DNS?)

The DNS worked fine , falcoctl does not have shell the status is waiting, I did curl from another pod though, Having http_proxy as ENV variable and from the pod it was able to curl the falcoctl index

This is very strange behavior, we can see with openssl command that there is no issue with proxy too....

Do you confirm the error comes from the falcoctl-artifact-install container and not from the falcoctl-artifact-follow container?

I'm asking because there are two different env configs (I know it is a bit cumbersome):

  • falcoctl.artifact.install.env
  • falcoctl.artifact.follow.env

Both should be set if you enable both features.

leogr avatar Jun 12 '25 14:06 leogr

Checked with http://myproxy.com:8080/ getting same

If you meant that the proxy isn't resolving or responding as expected, the issue isn't in falcoctl. For example, if you run from within the falcoctl container: curl -v --proxy https://myproxy.com:8080 https://example.com/ and you get the same problem, likely the root cause is in the way the container reaches the proxy (eg. bad DNS?)

The DNS worked fine , falcoctl does not have shell the status is waiting, I did curl from another pod though, Having http_proxy as ENV variable and from the pod it was able to curl the falcoctl index This is very strange behavior, we can see with openssl command that there is no issue with proxy too....

Do you confirm the error comes from the falcoctl-artifact-install container and not from the falcoctl-artifact-follow container?

I'm asking because there are two different env configs (I know it is a bit cumbersome):

  • falcoctl.artifact.install.env
  • falcoctl.artifact.follow.env

Both should be set if you enable both features.

Yes, it is coming from the falcoctl-artifact-install container not from falcoctl-artifact-follow container both are set the same env,

@FYI, Logs from falcoctl.artifact.follow.env

Failed to load logs: container "falcoctl-artifact-follow" in pod "falco-68gpc" is waiting to start: PodInitializing Reason: BadRequest (400)

Su-HackZ avatar Jun 13 '25 05:06 Su-HackZ

Yes, it is coming from the falcoctl-artifact-install container not from falcoctl-artifact-follow container both are set the same env,

@fyi, Logs from falcoctl.artifact.follow.env

Failed to load logs: container "falcoctl-artifact-follow" in pod "falco-68gpc" is waiting to start: PodInitializing Reason: BadRequest (400)

Got it. Thank you for confirming. Unfortunately, I've no idea how to reproduce the issue at the moment. I will try to dig into it once I get some spare cycles. If you have any suggestions, please let us know!

leogr avatar Jun 13 '25 14:06 leogr

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana avatar Sep 11 '25 16:09 poiana

/remove-lifecycle stale

leogr avatar Sep 15 '25 12:09 leogr