dashboard Kong pod doesn't start

What happened?

Issue

I've seen other issues related to this https://github.com/kubernetes/dashboard/issues/8909 https://github.com/kubernetes/dashboard/issues/8765. https://github.com/kubernetes/dashboard/issues/8909 is similar to my issue. I faced this issue previously which got fixed by upgrading docker from 4.28.0 to 4.29.0 and reinstalling dashboard.

Now I had to reinstall docker for different reason and I'm facing the issue again. I see some comments mentioning 8444 port is being used by minikube. In my case 8444 port is free and I dont have minikube. But I have tomcat on 8443.

Workaround

Only workaround is to delete the kong pod which makes the pod start successfully. But after restarting windows the issue reappears.

Environment

OS: Windows Docker Desktop version: 4.38.0 (181591) I access dashboard with https://localhost:32443/ instead of port forward. Maybe I enabled nodeport. I don't remember.

>kubectl logs kubernetes-dashboard-kong-678c76c548-gkxdr -n kubernetes-dashboard
Defaulted container "proxy" out of: proxy, clear-stale-pid (init)
2025/09/17 03:54:18 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
2025/09/17 03:54:18 [notice] 1#0: [lua] init.lua:791: init(): [request-debug] token for request debugging: 7e6103b3-2a41-47a8-a04e-baad72f87342
2025/09/17 03:54:18 [emerg] 1#0: bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
nginx: [emerg] bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
2025/09/17 03:54:18 [notice] 1#0: try again to bind() after 500ms
2025/09/17 03:54:18 [emerg] 1#0: bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
nginx: [emerg] bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
2025/09/17 03:54:18 [notice] 1#0: try again to bind() after 500ms
2025/09/17 03:54:18 [emerg] 1#0: bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
nginx: [emerg] bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
2025/09/17 03:54:18 [notice] 1#0: try again to bind() after 500ms
2025/09/17 03:54:18 [emerg] 1#0: bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
nginx: [emerg] bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
2025/09/17 03:54:18 [notice] 1#0: try again to bind() after 500ms
2025/09/17 03:54:18 [emerg] 1#0: bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
nginx: [emerg] bind() to unix:/kong_prefix/sockets/we failed (98: Address already in use)
2025/09/17 03:54:18 [notice] 1#0: try again to bind() after 500ms
2025/09/17 03:54:18 [emerg] 1#0: still could not bind()
nginx: [emerg] still could not bind()

What did you expect to happen?

Kubernetes dashboard should start and be accessible

How can we reproduce it (as minimally and precisely as possible)?

Issue is happening after installing kubernetes dashboard via helm.

Anything else we need to know?

No response

What browsers are you seeing the problem on?

No response

Kubernetes Dashboard version

7.13.0

Kubernetes version

Client Version: v1.31.4 Kustomize Version: v5.4.2 Server Version: v1.31.4

Dev environment

No response

Sep 17 '25 08:09 AndeYashwanth

Hi @AndeYashwanth,

Based on the error logs you've provided, this appears to be a socket binding issue where Kong is unable to bind to unix:/kong_prefix/sockets/we because it's already in use.

Here are some steps you can try to resolve this:

Check for existing processes: Before starting Kong, ensure no other process is using the socket file. You can check this by running:
```
lsof | grep "/kong_prefix/sockets/we"
```
Clean up stale socket files: Manually remove any lingering socket files before starting/restarting the pod:
```
kubectl exec -it <kong-pod-name> -n kubernetes-dashboard -- rm -f /kong_prefix/sockets/we
```
Or if accessible from the host:
```
rm -f /path/to/kong_prefix/sockets/we
```
Check socket file permissions: If the error persists, verify the permissions on the socket directory:
```
kubectl exec -it <kong-pod-name> -n kubernetes-dashboard -- ls -la /kong_prefix/sockets/
```
The Kong process should have write permissions to create socket files in this directory.
Container restart with cleanup: Try deleting the pod (as you mentioned works) but also ensure the underlying socket files are cleaned up before the new pod starts.

This issue often occurs when:

Previous Kong processes didn't shut down cleanly
Socket files weren't properly cleaned up during container restarts
Permission issues prevent proper socket file management

Since you mentioned this happens after Windows restarts, it's likely related to how Docker Desktop handles container cleanup on Windows. The socket files may be persisting between restarts.

Let me know if this helps resolve the issue!

Sep 24 '25 18:09 Sayeem3051

@Sayeem3051 I'm on windows host, so I can't run 1. Since the pod doesn't start due to the bind error, I cant exec into the pod to run 2 and 3.

>kubectl exec -it kubernetes-dashboard-kong-678c76c548-nbcz7 -n kubernetes-dashboard -- rm -f /kong_prefix/sockets/we
Defaulted container "proxy" out of: proxy, clear-stale-pid (init)
error: Internal error occurred: unable to upgrade connection: container not found ("proxy")

How can I ensure socket files are cleaned up after the pod is deleted? Also, the pod is recreated automatically once I delete it, so I'm not sure how I can check something before the new pod starts. This is what I get when I run 3 after new pod is started successfully.

>kubectl exec -it kubernetes-dashboard-kong-678c76c548-m87nb -n kubernetes-dashboard -- ls -la /kong_prefix/sockets/
Defaulted container "proxy" out of: proxy, clear-stale-pid (init)
total 8
drwxr-xr-x  2 kong kong 4096 Sep 25 17:48 .
drwxrwxrwx 12 root root 4096 Sep 25 17:48 ..
srw-rw-rw-  1 kong kong    0 Sep 25 17:48 we

To add to the troubleshooting, I've also tried restarting the pod and docker but it didn't fix the issue.

Sep 25 '25 18:09 AndeYashwanth

@AndeYashwanth #9955 I got the same issue yesterday, bump kong to 3.9 fixed it for me.

Sep 26 '25 17:09 jackthenewbie

@jackthenewbie Thanks. Currently kong 3.8 is in use. I've ran the upgrade command workaround. I'll restart windows later and see if that fixes the issue.

Sep 26 '25 17:09 AndeYashwanth

I can confirm that the issue is resolved after uppdating kong image to 3.9.0

Sep 28 '25 08:09 AndeYashwanth