Get error in local environment ,

[s5364733]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• Version [e.g. 2.4.4]
• Kubernetes version [e.g. 1.21.10]
• Package version [e.g. Helm 3.2, carvel-imgpkg 0.28.0]

Additional context Add any other context about the problem here.

[s5364733]

There is no way to use it at all. I need to refresh all the time, and suddenly it times out.

[s5364733]

[absoludity]

Hi @s5364733 . Can you watch the output of kubectl -n kubeapps get pods while you are experiencing those issues? A 502 bad gateway usually means that the frontend (nginx) is unable to forward the requests because the service it's trying to forward to is unavailable. Most likely the kubeapps-apis service is being constantly rebooted due to a lack of resources (not enough memory available, being the most obvious).

Note that if you are running this locally, you need a decent amount of grunt on your local machine (32GB ram), and even then, use the site/content/docs/latest/reference/manifests/kubeapps-local-dev-values.yaml to ensure you only have one of each service running (in prod you'd want multiple, but on a local development environment, it's too much for the one machine).

[s5364733]

Hi @s5364733 . Can you watch the output of kubectl -n kubeapps get pods while you are experiencing those issues? A 502 bad gateway usually means that the frontend (nginx) is unable to forward the requests because the service it's trying to forward to is unavailable. Most likely the kubeapps-apis service is being constantly rebooted due to a lack of resources (not enough memory available, being the most obvious).

Note that if you are running this locally, you need a decent amount of grunt on your local machine (32GB ram), and even then, use the site/content/docs/latest/reference/manifests/kubeapps-local-dev-values.yaml to ensure you only have one of each service running (in prod you'd want multiple, but on a local development environment, it's too much for the one machine).

[s5364733]

[s5364733]

in prod you'd want multiple, but on a local development environment, it's too much for the one machine

All services are started normally

[absoludity]

Thanks for the extra info.

in prod you'd want multiple, but on a local development environment, it's too much for the one machine

All services are started normally

Not according to both your screenshots? You've got two kubeapps-internal-kubeappsapis-* pods and one postgresql pod, all showing the "Running" state, but none of which is showing ready? (all show 0/1 ready)

This means that none of those pods are in a ready state (failing their ready checks). It would be worth taking a look at the k8s documentation for how to get more info about the ready checks. My guess, as before (and strengthened by the number of restarts you are seeing) is that there's just not enough resources (memory, EDIT: or CPU) available and so k8s keeps killing pods to start others etc. (for eg., your postgresql pod has 81 restarts in the 23 hours, over 3 per hour).

Note: free -mh shows you how much free mem you've got on your machine, not how much you've allocated for use with docker necessarily (varies based on system).

As mentioned earlier, I would use the local values that I pointed to so that you're not running two pods when you only need one.

You might find that the output of kubectl --namespace kubeapps describe pod kubeapps-postgresql-0 gives you more info about why the pod is not ready (see the checks). Or paste it here, with the logs for that pod, out of interest. Cheers.

[s5364733]

Thanks for the extra info.

in prod you'd want multiple, but on a local development environment, it's too much for the one machine

All services are started normally

Not according to both your screenshots? You've got two kubeapps-internal-kubeappsapis-* pods and one postgresql pod, all showing the "Running" state, but none of which is showing ready? (all show 0/1 ready)

This means that none of those pods are in a ready state (failing their ready checks). It would be worth taking a look at the k8s documentation for how to get more info about the ready checks. My guess, as before (and strengthened by the number of restarts you are seeing) is that there's just not enough resources (memory, EDIT: or CPU) available and so k8s keeps killing pods to start others etc. (for eg., your postgresql pod has 81 restarts in the 23 hours, over 3 per hour).

Note: free -mh shows you how much free mem you've got on your machine, not how much you've allocated for use with docker necessarily (varies based on system).

As mentioned earlier, I would use the local values that I pointed to so that you're not running two pods when you only need one.

You might find that the output of kubectl --namespace kubeapps describe pod kubeapps-postgresql-0 gives you more info about why the pod is not ready (see the checks). Or paste it here, with the logs for that pod, out of interest. Cheers.

Name: kubeapps-postgresql-0 Namespace: kubeapps Priority: 0 Service Account: default Node: minikube/192.168.39.33 Start Time: Tue, 19 Sep 2023 11:03:12 +0800 Labels: app.kubernetes.io/component=primary app.kubernetes.io/instance=kubeapps app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=postgresql controller-revision-hash=kubeapps-postgresql-76c6bbd8c9 helm.sh/chart=postgresql-12.10.0 statefulset.kubernetes.io/pod-name=kubeapps-postgresql-0 Annotations: Status: Running IP: 10.244.0.235 IPs: IP: 10.244.0.235 Controlled By: StatefulSet/kubeapps-postgresql Containers: postgresql: Container ID: docker://191c5ac02958972248103e3fdd1ad3a3cde3a8ca74acc08873ae0e21b7f76b76 Image: docker.io/bitnami/postgresql:15.4.0-debian-11-r10 Image ID: docker-pullable://bitnami/postgresql@sha256:86c140fd5df7eeb3d8ca78ce4503fcaaf0ff7d2e10af17aa424db7e8a5ae8734 Port: 5432/TCP Host Port: 0/TCP SeccompProfile: RuntimeDefault State: Running Started: Wed, 20 Sep 2023 12:16:45 +0800 Last State: Terminated Reason: Error Exit Code: 137 Started: Wed, 20 Sep 2023 12:14:22 +0800 Finished: Wed, 20 Sep 2023 12:16:35 +0800 Ready: True Restart Count: 97 Requests: cpu: 250m memory: 256Mi Liveness: exec [/bin/sh -c exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6 Readiness: exec [/bin/sh -c -e exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432 [ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ] ] delay=5s timeout=5s period=10s #success=1 #failure=6 Environment: BITNAMI_DEBUG: false POSTGRESQL_PORT_NUMBER: 5432 POSTGRESQL_VOLUME_DIR: /bitnami/postgresql PGDATA: /bitnami/postgresql/data POSTGRES_PASSWORD: <set to the key 'postgres-password' in secret 'kubeapps-postgresql'> Optional: false POSTGRES_DATABASE: assets POSTGRESQL_ENABLE_LDAP: no POSTGRESQL_ENABLE_TLS: no POSTGRESQL_LOG_HOSTNAME: false POSTGRESQL_LOG_CONNECTIONS: false POSTGRESQL_LOG_DISCONNECTIONS: false POSTGRESQL_PGAUDIT_LOG_CATALOG: off POSTGRESQL_CLIENT_MIN_MESSAGES: error POSTGRESQL_SHARED_PRELOAD_LIBRARIES: pgaudit Mounts: /dev/shm from dshm (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vfpw (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-9vfpw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Warning BackOff 33m (x104 over 77m) kubelet Back-off restarting failed container postgresql in pod kubeapps-postgresql-0_kubeapps(fa0182c9-e300-4311-88c0-f5ceccfce91b) Warning Unhealthy 23m (x435 over 168m) kubelet Readiness probe failed: command "/bin/sh -c -e exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432\n[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]\n" timed out Warning Unhealthy 18m (x156 over 164m) kubelet Liveness probe failed: command "/bin/sh -c exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432" timed out Warning Unhealthy 13m (x5 over 58m) kubelet Readiness probe failed: cannot exec in a stopped state: unknown

[s5364733]

How can I allocate more memory to docker? Or do you have any better solution to solve this problem?

[s5364733]

docker info : Client: Version: 24.0.5 Context: default Debug Mode: false

Server: Containers: 9 Running: 2 Paused: 0 Stopped: 7 Images: 13 Server Version: 24.0.5 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: syslog Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: runc version: init version: Security Options: seccomp Profile: builtin Kernel Version: 5.15.90.1-microsoft-standard-WSL2 Operating System: Ubuntu 20.04.6 LTS OSType: linux Architecture: x86_64 CPUs: 10 Total Memory: 15.28GiB Name: Jackliang ID: 26d9706e-e3b4-437d-b407-86cbc2782ff9 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Registry Mirrors: https://ung2thfc.mirror.aliyuncs.com/ Live Restore Enabled: false

[s5364733]

Thanks for the extra info.

in prod you'd want multiple, but on a local development environment, it's too much for the one machine

All services are started normally

Not according to both your screenshots? You've got two kubeapps-internal-kubeappsapis-* pods and one postgresql pod, all showing the "Running" state, but none of which is showing ready? (all show 0/1 ready) This means that none of those pods are in a ready state (failing their ready checks). It would be worth taking a look at the k8s documentation for how to get more info about the ready checks. My guess, as before (and strengthened by the number of restarts you are seeing) is that there's just not enough resources (memory, EDIT: or CPU) available and so k8s keeps killing pods to start others etc. (for eg., your postgresql pod has 81 restarts in the 23 hours, over 3 per hour). Note: free -mh shows you how much free mem you've got on your machine, not how much you've allocated for use with docker necessarily (varies based on system). As mentioned earlier, I would use the local values that I pointed to so that you're not running two pods when you only need one. You might find that the output of kubectl --namespace kubeapps describe pod kubeapps-postgresql-0 gives you more info about why the pod is not ready (see the checks). Or paste it here, with the logs for that pod, out of interest. Cheers.

Name: kubeapps-postgresql-0 Namespace: kubeapps Priority: 0 Service Account: default Node: minikube/192.168.39.33 Start Time: Tue, 19 Sep 2023 11:03:12 +0800 Labels: app.kubernetes.io/component=primary app.kubernetes.io/instance=kubeapps app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=postgresql controller-revision-hash=kubeapps-postgresql-76c6bbd8c9 helm.sh/chart=postgresql-12.10.0 statefulset.kubernetes.io/pod-name=kubeapps-postgresql-0 Annotations: Status: Running IP: 10.244.0.235 IPs: IP: 10.244.0.235 Controlled By: StatefulSet/kubeapps-postgresql Containers: postgresql: Container ID: docker://191c5ac02958972248103e3fdd1ad3a3cde3a8ca74acc08873ae0e21b7f76b76 Image: docker.io/bitnami/postgresql:15.4.0-debian-11-r10 Image ID: docker-pullable://bitnami/postgresql@sha256:86c140fd5df7eeb3d8ca78ce4503fcaaf0ff7d2e10af17aa424db7e8a5ae8734 Port: 5432/TCP Host Port: 0/TCP SeccompProfile: RuntimeDefault State: Running Started: Wed, 20 Sep 2023 12:16:45 +0800 Last State: Terminated Reason: Error Exit Code: 137 Started: Wed, 20 Sep 2023 12:14:22 +0800 Finished: Wed, 20 Sep 2023 12:16:35 +0800 Ready: True Restart Count: 97 Requests: cpu: 250m memory: 256Mi Liveness: exec [/bin/sh -c exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6 Readiness: exec [/bin/sh -c -e exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432 [ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ] ] delay=5s timeout=5s period=10s #success=1 #failure=6 Environment: BITNAMI_DEBUG: false POSTGRESQL_PORT_NUMBER: 5432 POSTGRESQL_VOLUME_DIR: /bitnami/postgresql PGDATA: /bitnami/postgresql/data POSTGRES_PASSWORD: <set to the key 'postgres-password' in secret 'kubeapps-postgresql'> Optional: false POSTGRES_DATABASE: assets POSTGRESQL_ENABLE_LDAP: no POSTGRESQL_ENABLE_TLS: no POSTGRESQL_LOG_HOSTNAME: false POSTGRESQL_LOG_CONNECTIONS: false POSTGRESQL_LOG_DISCONNECTIONS: false POSTGRESQL_PGAUDIT_LOG_CATALOG: off POSTGRESQL_CLIENT_MIN_MESSAGES: error POSTGRESQL_SHARED_PRELOAD_LIBRARIES: pgaudit Mounts: /dev/shm from dshm (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vfpw (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-9vfpw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Warning BackOff 33m (x104 over 77m) kubelet Back-off restarting failed container postgresql in pod kubeapps-postgresql-0_kubeapps(fa0182c9-e300-4311-88c0-f5ceccfce91b) Warning Unhealthy 23m (x435 over 168m) kubelet Readiness probe failed: command "/bin/sh -c -e exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432\n[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]\n" timed out Warning Unhealthy 18m (x156 over 164m) kubelet Liveness probe failed: command "/bin/sh -c exec pg_isready -U "postgres" -d "dbname=assets" -h 127.0.0.1 -p 5432" timed out Warning Unhealthy 13m (x5 over 58m) kubelet Readiness probe failed: cannot exec in a stopped state: unknown

Looks like a network problem？

[antgamdia]

Can't reproduce it locally, did you manage to get it sorted out finally?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.