Possible issue in K8s cluster with cilium

[tschale]

What happened?

Setup: kubermatic/kubeone on Hetzner VMs, with cilium addon enabled.

Container from gefyra run can't resolve database connection (K8s service in the cluster). Env variables are all correctly taken from the specified deployment.

A traceroute from the container to the database host shows that it can resolve the FQDN, as well as the IP of the service and seems to reach an IP address inside the cluster. Although I wasn't able to match that IP address to a pod, service or node.

I am of the opinion that it did work at some time in that cluster. However, I can't say for sure, it could also be that I wrongly remember that...

What did you expect to happen?

Container from gefyra run to work without issue, more specific to be able to reach the database host.

Please provide the output of gefyra check.

[INFO] Checking Docker client. [INFO] Docker client: Ok [INFO] Checking availability of Gefyra Cargo image... [INFO] Gefyra Cargo: Available [INFO] Docker: Ok [INFO] Checking Kubernetes connection. [INFO] Kubernetes: Ok [INFO] Gefyra client version: 1.1.2

How can we reproduce it (as minimally and precisely as possible)?

One of our project clusters, get in contact with me for more infos.

What Kubernetes setup are you working with?

$ kubectl version
# paste output here

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:51:25Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux gutschi-ThinkPad-P14s-Gen-2i 6.2.0-32-generic #32~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 18 10:40:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Anything else we need to know?

No response

[Schille]

I investigated this issue and came across cilium/cilium#27758 It seems to be an issue with Cilium at the moment. I'll keep you posted.

[RichardSufliarsky]

@tschale try to look at this: https://github.com/cilium/cilium/issues/27758#issuecomment-1718282433

[Schille]

FWIW: Patching ConfigMap cilium-config with kube-proxy-replacement=strict worked here.

[Schille]

@SteinRobert I'd close this. It is very specific and we can't do anything about it.