Superuser Creation: `backend-server` does not have a host assigned

[ifTaylor]

Actions before raising this issue

• [X] I searched the existing issues and did not find anything similar.
• [X] I read/searched the docs

Steps to Reproduce

1. Changes made to values.yaml
   • permissionFix: enabled: false
   • traefik: enabled: true
2. Changes made to Chart.yaml
   • name: traefik version: 27.0.0
3. run $ helm upgrade -n cvat latest -i --create-namespace ./helm-chart -f ./helm-chart/values.yaml
4. run "How to create superuser?"

Expected Behavior

Superuser made and cvat-backend container in backend-server pod not at a Waiting state.

Possible Solution

No response

Context

Output when trying to create superuser: Error from server (BadRequest): pod latest-backend-server-78b9bfb9f9-gpq98 does not have a host assigned

Pod status:

$ kubectl get pods -n cvat

NAME                                                     READY   STATUS             RESTARTS       AGE
latest-backend-initializer-r14-c25m7                     0/1     Completed          0              118m
latest-backend-initializer-r16-2qtdm                     0/1     Completed          0              63m
latest-backend-initializer-r18-x4dwm                     0/1     Completed          0              43m
latest-backend-server-78b9bfb9f9-gpq98                   0/1     Pending            0              43m
latest-backend-utils-5644445b6-9mlmp                     0/1     Pending            0              43m
latest-backend-worker-analyticsreports-5ccfc8c5c-8g98w   1/1     Running            0              4h2m
latest-backend-worker-annotation-7b69cd8f8f-dw4xx        0/1     Pending            0              43m
latest-backend-worker-export-f8cf55747-4lqpk             0/1     Pending            0              43m
latest-backend-worker-export-f8cf55747-5bvbx             0/1     Pending            0              43m
latest-backend-worker-import-56445b6f77-9p5r9            0/1     Pending            0              43m
latest-backend-worker-import-56445b6f77-v8rch            0/1     Pending            0              43m
latest-backend-worker-qualityreports-7f6c778f6c-jrbjx    1/1     Running            0              4h2m
latest-backend-worker-webhooks-68bfdd87dd-bv2j5          1/1     Running            0              4h2m
latest-clickhouse-shard0-0                               1/1     Running            0              4h17m
latest-frontend-6f54b65494-c69k7                         1/1     Running            0              4h17m
latest-grafana-56b7ddfc97-wc6l2                          1/1     Running            0              4h9m
latest-kvrocks-0                                         1/1     Running            0              4h17m
latest-opa-55655d79b4-6mjph                              1/1     Running            0              4h17m
latest-postgresql-0                                      1/1     Running            0              4h2m
latest-redis-master-0                                    1/1     Running            0              4h17m
latest-traefik-57c4b85cfb-5ctfl                          1/1     Running            0              4h17m
latest-vector-0                                          0/1     CrashLoopBackOff   55 (56s ago)   4h17m

I've had this working before with a different CVAT version and the CrashLoopBackOff was resolved when creating the superuser.

Environment

- Using Kubernetes.

[FranLucchini]

Hello, we've have a similar issue if not the same issue. Out of curiosity, where are you deploying your k8s cluster and how many resources does it have?

We are currently using a GCP K8s cluster and the backend pods send an error that there are not enough resources for initialization.

[ifTaylor]

@FranLucchini

where are you deploying your k8s cluster

I am using AKS with four nodes, although only one is being allocated to the running pods.

Our deployments do sound similar. I have not been able to work more on this, but will be picking this back up next sprint. Have you found any solution?

[mathijscarlu]

@ifTaylor I'm also deploying on AKS. The reason the createsuperuser command is failing, is that you are basically trying to execute a command in a pod that is not there yet (latest-backend-server-78b9bfb9f9-gpq98 in your case, which is in Pending state).

For me, the reason it was stuck in Pending state was the fact that PVCs where created with storageClass azure disk, which does not support ReadWriteMany access mode. The azure file class does, however.

[ifTaylor]

@mathijscarlu Thanks for the insight! That let me to the right spot in the values.yaml.

By uncommenting the defaultStorage: elements and providing a storageClassName: in both cvat:backend: and kvrocks: I am producing this output.

$ kubectl get pods -n cvat

NAME                                                     READY   STATUS             RESTARTS         AGE
latest-backend-initializer-r20-sq7l9                     0/1     Completed          0                24m
latest-backend-initializer-r21-5bnrv                     0/1     Completed          0                21m
latest-backend-server-78b9bfb9f9-8tlnn                   0/1     Error              9 (5m12s ago)    22d
latest-backend-utils-5644445b6-9mlmp                     1/1     Running            0                23d
latest-backend-worker-analyticsreports-5ccfc8c5c-2795l   1/1     Running            0                16d
latest-backend-worker-annotation-7b69cd8f8f-dw4xx        1/1     Running            0                23d
latest-backend-worker-export-f8cf55747-4lqpk             1/1     Running            0                23d
latest-backend-worker-export-f8cf55747-5bvbx             1/1     Running            0                23d
latest-backend-worker-import-56445b6f77-9p5r9            1/1     Running            0                23d
latest-backend-worker-import-56445b6f77-v8rch            1/1     Running            0                23d
latest-backend-worker-qualityreports-7f6c778f6c-vcv8r    1/1     Running            0                16d
latest-backend-worker-webhooks-68bfdd87dd-m7688          1/1     Running            0                16d
latest-clickhouse-shard0-0                               1/1     Running            0                16d
latest-frontend-6f54b65494-jrdsq                         1/1     Running            0                16d
latest-grafana-56b7ddfc97-glcbx                          1/1     Running            0                16d
latest-kvrocks-0                                         1/1     Running            0                16d
latest-opa-55655d79b4-rg4jk                              1/1     Running            0                16d
latest-postgresql-0                                      1/1     Running            0                16d
latest-redis-master-0                                    1/1     Running            0                16d
latest-traefik-57c4b85cfb-x4dql                          1/1     Running            0                16d
latest-vector-0                                          0/1     CrashLoopBackOff   11 (2m25s ago)   16d

Still not able to create superuser though.

$ kubectl exec -it --namespace $HELM_RELEASE_NAMESPACE $BACKEND_POD_NAME -c cvat-backend -- python manage.py createsuperuser

>> error: unable to upgrade connection: container not found ("cvat-backend")

[mathijscarlu]

@ifTaylor Feel free to check out my issue linked above and verify if you are having the same issue. If so, I have provided a PR that adds a cvat.backend.initSecretKey.enabled flag that should solve your problem if set to true.