Gateway not working from inside kubernetes cluster

[tahvane1]

Is there an existing issue for this?

• [x] I have searched the existing issues

Version

equal or higher than v1.17.4 and lower than v1.18.0

What happened?

I'm running netkit and native routing. BPF masguerading but no nat for these IPs. I get this drop where I think source labels are messed up as I cant' figure out network policy to allow it.

How can we reproduce the issue?

create gateway http route and try to connect from inside cluster

Cilium Version

1.17.4

Kernel Version

6.13.12

Kubernetes Version

v1.32.4+rke2r1

Regression

No response

Sysdump

No response

Relevant log output

Anything else?

No response

Cilium Users Document

• [x] Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[youngnick]

Hi @tahvane1, I'd recommend reviewing https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/#cilium-s-ingress-config-and-ciliumnetworkpolicy and the example at https://docs.cilium.io/en/stable/network/servicemesh/ingress-and-network-policy/#gs-ingress-and-network-policy to learn more about targeting Network Policy at the reserved:ingress identity.

In particular, the example of allowing Ingress traffic when using a default deny seems like it might be the most useful. https://docs.cilium.io/en/stable/network/servicemesh/ingress-and-network-policy/#default-deny-ingress-policy

[tahvane1]

ExternalTrafficPolicy is cluster for gateway. I have tried all kinds of policies for ingress. For example:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-ingress-egress
  annotations:
    argocd.argoproj.io/hook: PreSync
specs:
- endpointSelector:
    matchExpressions:
    - key: reserved:ingress
      operator: Exists
  egress:
  - toEntities:
    - cluster
  - toCIDR:
    - 172.16.1.4/32
  ingress:
  - fromEntities:
    - world
    - cluster

And I have spend quite many hours to trying to solve this (as it is really pain in the ass trying to support services with internal/external names at the same time.

[tahvane1]

Envoy is returning access denied

git@gitlab-toolbox-54fdb94c64-k64fx:/$ curl -vv -k -X POST https://gitlab.XXX --resolve gitlab.XXX:443:172.16.1.4

• Added gitlab.XXX:443:172.16.1.4 to DNS cache
• Hostname gitlab.XXX was found in DNS cache
• Trying 172.16.1.4:443...
• Connected to gitlab.XXX (172.16.1.4) port 443 (#0)
• ALPN: offers h2,http/1.1
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN: server accepted h2
• Server certificate:
• subject: CN=*.XXX
• start date: Jun 6 17:07:17 2025 GMT
• expire date: Sep 4 17:07:16 2025 GMT
• issuer: C=US; O=Let's Encrypt; CN=R11
• SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
• using HTTP/2
• h2h3 [:method: POST]
• h2h3 [:path: /]
• h2h3 [:scheme: https]
• h2h3 [:authority: gitlab.XXX]
• h2h3 [user-agent: curl/7.88.1]
• h2h3 [accept: /]
• Using Stream ID: 1 (easy handle 0x62b04b51e7a0)

POST / HTTP/2 Host: gitlab.XXX user-agent: curl/7.88.1 accept: /

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• old SSL session ID is stale, removing < HTTP/2 403 < content-length: 15 < content-type: text/plain < date: Mon, 23 Jun 2025 08:43:23 GMT < server: envoy < Access denied

[youngnick]

The Network Policy you supplied looks like it should allow traffic for the reserved:ingress identity, but remember that if you have a deny-all policy, you'll also need to allow traffic to egress from the source Pod to the reserved:ingress identity. If you don't have that Policy, then that will also be enforced in Envoy, but will be reported as you've listed.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[github-actions[bot]]

This issue has not seen any activity since it was marked stale. Closing.

[tahvane1]

We have removed nat and now we get this kind of issue inside cluster Nov 17 17:37:00.402: sentry/sentry-worker-977dfc45f-zh2m9:54064 (ingress) -> kube-system/cilium-gateway-public-gateway:443 (ID:16777712) http-request DROPPED (HTTP/2 POST https://sentry.confidently.fi/api/1/envelope/)

ID 16777712 is fqdn sentry.confidently.fi

[tahvane1]

I added also port 80 and redirect from there to port 443. This part works also from inside cluster so it can't be routing/hairpin issue. I also tried without host firewall and connections are still dropped.

[tahvane1]

I added service behind port 80 and now it gets access denied similarly to https. gitlab/gitlab-webservice-default-79655d87d6-26cds:48322 (ingress) -> kube-system/cilium-gateway-public-gateway:80 (world) http-request DROPPED (HTTP/1.1 GET http://abc.confidently.fi/)

[tahvane1]

I tested this also with ingress and similar issue.