Failure to start cluster with `unmanaged: true` on latest versions

[jonathon2nd]

I opened this issue when I attempted with previous version, but even latest versions do not work, so updated title.

Report

Creation of cluster with unmanaged: true fails, mongodb nodes bootloop.

Operator continues to log the following repeatedly as the mongo node bootloop. INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "4071c15b-9595-443f-bb20-5705204cbd3d", "replset": "rs0"}

More about the problem

I am attempting to follow this guide to migrate one of our legacy mongodb without downtime.

SSL off as the internal db we are using does not have that on, I will turn it on later with short downtime after migration is successful.

Yaml

---
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: example-mongodb
  namespace: mongodb
spec:
  allowUnsafeConfigurations: true
  unsafeFlags:
    tls: true
  unmanaged: true

  crVersion: 1.15.0
  image: percona/percona-server-mongodb:4.4.24
  tls:
    mode: disabled
  replsets:
    - affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      name: rs0
      size: 3
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 100Gi
      expose:
        enabled: false
        exposeType: LoadBalancer
  secrets:
    users: users
  sharding:
    enabled: false
  backup:
    enabled: false
    pitr:
      enabled: false
  upgradeOptions:
    apply: disabled
    schedule: 0 2 * * *

Steps to reproduce

1. Deploy yaml as described, wait for failure.

Versions

1. Kubernetes: v1.26.13 +rke2r1
2. Operator: First tried with 1.16.1, upgraded to 1.16.3, no change

Anything else?

The DB I am attempting to migrate requires a target of 4.4, hence the cr version selection.

The deployment works without unmanaged: true

Two pod logs from start up to bootloop example-mongodb-rs0-0_mongod.log example-mongodb-rs0-1_mongod.log

Operator logs:

2024-07-24T18:21:17.244954170-04:00 2024-07-24T22:21:17.244Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "replset": "rs0"}
2024-07-24T18:21:17.519518649-04:00 2024-07-24T22:21:17.518Z	INFO	Created a new mongo key	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "KeyName": "example-mongodb-mongodb-keyfile"}
2024-07-24T22:21:17.535Z	INFO	Created a new mongo key	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "KeyName": "example-mongodb-mongodb-encryption-key"}
2024-07-24T18:21:17.699208811-04:00 2024-07-24T22:21:17.698Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:17.751281423-04:00 2024-07-24T22:21:17.750Z	INFO	add new job	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "name": "ensure-version/mongodb/example-mongodb", "schedule": "0 2 * * *"}
2024-07-24T18:21:17.779567104-04:00 2024-07-24T22:21:17.779Z	INFO	Cluster state changed	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "previous": "", "current": "initializing"}
2024-07-24T18:21:17.816437308-04:00 2024-07-24T22:21:17.815Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "a48c61c2-ac90-4f17-b581-cf8d1a3c5dbe", "replset": "rs0"}
2024-07-24T18:21:18.096219954-04:00 2024-07-24T22:21:18.095Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "a48c61c2-ac90-4f17-b581-cf8d1a3c5dbe", "replset": "rs0", "size": 3, "pods": 
1}
2024-07-24T18:21:22.815267111-04:00 2024-07-24T22:21:22.814Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "97ee1bca-d292-44c0-ae9b-4adf1dc2570d", "replset": "rs0"}
2024-07-24T22:21:23.071Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "97ee1bca-d292-44c0-ae9b-4adf1dc2570d", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:28.187110713-04:00 2024-07-24T22:21:28.186Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "9be35d66-6137-4d9d-aa13-2d98d1658fc2", "replset": "rs0"}
2024-07-24T18:21:28.489985716-04:00 2024-07-24T22:21:28.489Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "9be35d66-6137-4d9d-aa13-2d98d1658fc2", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:33.639456067-04:00 2024-07-24T22:21:33.639Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10c49def-f0d8-4766-96e6-adf1c185f949", "replset": "rs0"}
2024-07-24T22:21:33.943Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10c49def-f0d8-4766-96e6-adf1c185f949", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T22:21:39.363Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "21bb72e2-b804-4246-9ce7-45b87bce823d", "replset": "rs0"}
2024-07-24T22:21:39.630Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "21bb72e2-b804-4246-9ce7-45b87bce823d", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T22:21:45.632Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "e8703952-b61d-47a0-9193-27b57146df16", "replset": "rs0"}
2024-07-24T22:21:45.919Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "e8703952-b61d-47a0-9193-27b57146df16", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:46.084Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "14129a56-3d0e-47fc-981a-7c86b951d285", "replset": "rs0"}
2024-07-24T22:21:46.405Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "14129a56-3d0e-47fc-981a-7c86b951d285", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:51.312Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "c667ba1e-1d82-4850-b17f-759d61e0e4dd", "replset": "rs0"}
2024-07-24T22:21:51.619Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "c667ba1e-1d82-4850-b17f-759d61e0e4dd", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:57.004Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fce820a1-c739-44e9-b578-47c9e3defcee", "replset": "rs0"}
2024-07-24T22:21:57.282Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fce820a1-c739-44e9-b578-47c9e3defcee", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:02.817Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "79d72aa6-f450-41f9-8997-e8869282f337", "replset": "rs0"}
2024-07-24T22:22:03.090Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "79d72aa6-f450-41f9-8997-e8869282f337", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:08.509Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fbcd6837-1e99-49c7-a801-37a6ab0d143d", "replset": "rs0"}
2024-07-24T22:22:08.795Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fbcd6837-1e99-49c7-a801-37a6ab0d143d", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:14.213Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10fdccca-4165-452b-8b56-043fdf63f8db", "replset": "rs0"}
2024-07-24T22:22:14.511Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10fdccca-4165-452b-8b56-043fdf63f8db", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:19.927Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "8a52d2f9-57a4-4a20-94a9-507980a8a66b", "replset": "rs0"}
2024-07-24T22:22:31.851Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "6e9ad316-6787-4cb8-8146-ed34accbea83", "replset": "rs0"}
2024-07-24T22:22:43.739Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "41260c8c-bd30-4b66-a827-5fbb1a2b7081", "replset": "rs0"}
2024-07-24T22:22:55.687Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "7c3eb48f-59d5-4892-a2d6-f75118109e5c", "replset": "rs0"}
2024-07-24T22:23:07.532Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "727a406f-4f22-4e05-b1f3-7d4856077402", "replset": "rs0"}
2024-07-24T22:23:19.398Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "18cf442f-00ab-4605-82e3-4eec2c330abd", "replset": "rs0"}
2024-07-24T22:23:31.221Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "d018f4e4-efa9-4749-95b4-fdd6e9929538", "replset": "rs0"}
2024-07-24T22:23:43.030Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "5ebc4aed-b855-4483-b404-58580864283d", "replset": "rs0"}
2024-07-24T22:23:54.899Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "8b2c2559-2804-4388-a798-732bd5986536", "replset": "rs0"}
2024-07-24T22:24:06.718Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "1680e0de-9a6f-4669-9b8c-8a88e98c5397", "replset": "rs0"}
2024-07-24T22:24:18.539Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "81a4d31a-1abc-4eaa-8b22-823200d9c57f", "replset": "rs0"}
2024-07-24T22:24:30.317Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "58e30cba-39a6-4d2a-88af-2cb736dce724", "replset": "rs0"}
2024-07-24T22:24:43.680Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller",