charts [bitnami/mongodb-sharded] Fail to create root user upon installation

Name and Version

bitnami/mongodb-sharded 6.0.1

What steps will reproduce the bug?

Install MongoDB sharded using a manual provisioner storage class with GlusterFS PVs
Provide these values to the chart: global.storageClass=<storageClass name>, volumePermissions.enabled=true, auth.enabled=true, auth.rootUser=root, auth.existingSecret=<existing-secret>. Alternatively, you can provide the rootPassword instead of the secret.
All pods get to running status, but if you check in the logs, configsvr-0 says Authentication Failed after restarting MongoDB. I noticed that there is a line right after creating the users that indicates the users failed to create: MongoServerError: not primary [direct: secondary] test> Uncaught. However, this is the entire error message, so it doesn't really help.

Are you using any custom parameters or values?

global.storageClass=storageClass name volumePermissions.enabled=true auth.enabled=true auth.rootUser=root auth.existingSecret=existing-secret or rootPassword=password

What is the expected behavior?

I expect for the chart to correctly make the root user so the pods can authenticate correctly after installed.

What do you see instead?

It seems the root user fails to be created due to an uncaught exception by Mongo. This is the entire logs of the mongodb-sharded-configsvr-0 pod:

mongodb-sharded-configsvr-0

18:49:55.97 INFO ==> Setting node as primary
mongodb 18:49:56.00
mongodb 18:49:56.01 Welcome to the Bitnami mongodb-sharded container
mongodb 18:49:56.01 Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 18:49:56.01 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 18:49:56.02
mongodb 18:49:56.02 INFO ==> ** Starting MongoDB Sharded setup **
mongodb 18:49:56.05 INFO ==> Validating settings in MONGODB_* env vars...
mongodb 18:49:56.15 INFO ==> Initializing MongoDB Sharded...
mongodb 18:49:56.21 INFO ==> Deploying MongoDB Sharded from scratch...
MongoNetworkError: connect ECONNREFUSED 172.16.164.226:2701
mongodb 18:50:07.36 INFO ==> Creating users...
mongodb 18:50:07.36 INFO ==> Creating root user...
Current Mongosh Log ID: 62f3fde0d7ea7fdac5b96fb0
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.5.4
Using MongoDB: 6.0.0
Using Mongosh: 1.5.4

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
The server generated these startup warnings when booting
2022-08-10T18:49:58.117+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]
2022-08-10T18:49:58.118+00:00: vm.max_map_count is too low
------

------
Enable MongoDB's free cloud-based monitoring service, which will then receive and display metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------

MongoServerError: not primary [direct: secondary] test> Uncaught mongodb 18:50:09.14 INFO ==> Users created
mongodb 18:50:09.15 INFO ==> Writing keyfile for replica set authentication...
mongodb 18:50:09.18 INFO ==> Enabling authentication...
mongodb 18:50:09.20 INFO ==> Configuring MongoDB Sharded replica set...
mongodb 18:50:09.21 INFO ==> Stopping MongoDB...
mongodb 18:50:28.78 INFO ==> Configuring MongoDB primary node...: mongo-test-mongodb-configsvr-0.mongo-test-mongodb-headless.mongo-test.svc.cluster.local
MongoServerError: Authentication failed.

Additional information

This chart works perfectly fine using HostPath volumes. I haven't been able to test it on any other PV types. This might not be a problem with the chart itself, but the Uncaught Error makes me think something in the container is failing and not reporting the error well.

Aug 10 '22 19:08 jdholtz

Hi @jdholtz,

I haven't been able to reproduce the issue using the commands you provided except for the localpath storageclass. Is it possible that the volume is not entirely empty?

I see the message Deploying MongoDB Sharded from scratch... which is displayed right after checking the content of MONGODB_DATA_DIR/db, but maybe something else is directly under MONGODB_DATA_DIR and is causing the error .

    if is_dir_empty "$MONGODB_DATA_DIR/db"; then
        info "Deploying MongoDB Sharded from scratch..."

        ensure_dir_exists "$MONGODB_DATA_DIR/db"
        am_i_root && chown -R "$MONGODB_DAEMON_USER" "$MONGODB_DATA_DIR/db"

        mongodb_set_replicasetmode_conf

        if [[ "$MONGODB_SHARDING_MODE" =~ ^(configsvr|shardsvr)$ ]] && [[ "$MONGODB_REPLICA_SET_MODE" = "primary" ]]; then
            mongodb_sharded_initiate_svr_primary
        fi

        mongodb_start_bg
        mongodb_create_users
        mongodb_create_keyfile "$MONGODB_REPLICA_SET_KEY"
        mongodb_set_keyfile_conf
        mongodb_set_auth_conf
        mongodb_set_listen_all_conf
        mongodb_sharded_configure_replica_set
        mongodb_stop

Another possibility is that the node is not being initialized as the primary node, or another node is the primary. But during the first rolling deployment, node -0 should be assigned as primary:

  replicaset-entrypoint.sh: |-
    #!/bin/bash

    sleep 5

    . /liblog.sh

    # Perform adaptations depending on the host name
    if [[ $HOSTNAME =~ (.*)-0$ ]]; then
      info "Setting node as primary"
      export MONGODB_REPLICA_SET_MODE=primary
    else
      info "Setting node as secondary"
      export MONGODB_REPLICA_SET_MODE=secondary
      {{- if .Values.auth.usePasswordFile }}
      export MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD_FILE="$MONGODB_ROOT_PASSWORD_FILE"
      unset MONGODB_ROOT_PASSWORD_FILE
      {{- else }}
      export MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD="$MONGODB_ROOT_PASSWORD"
      unset MONGODB_ROOT_PASSWORD
      {{- end }}
    fi

    exec /entrypoint.sh /run.sh

Aug 11 '22 09:08 migruiz4

Hi. Thanks for responding quickly.

I am able to get the chart to work perfectly fine with Host path volumes and I have tested with NFS volumes with no issues. It might be an issue with GlusterFS, but I have gotten it the database to successfully start on Gluster once every ten times.

I do ensure that there is no data within the MongoDB directories, as I make sure to delete the everything inside of the directories before reinstalling.

As for the primary node thing, I have realized that the error I am getting is MongoServerError: not primary [direct: secondary] test> Uncaught which indicates something is wrong with configuring the primary node (although it says above it is being configured as the primary node at the start of the logs). However, when it succeeds, it displays this instead:

mongo-test-mongodb-configsvr [direct: primary] test> {
  ok: 1,
  lastCommittedOpTime: Timestamp({ t: 1660226930, i: 1 }),
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1660226931, i: 3 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1660226931, i: 3 })
}

Aug 11 '22 14:08 jdholtz

Hi @jdholtz,

I may not be able to help you further if the issue only happens when using GlusterFS and not other storage systems.

If you would like to troubleshoot the issue deeper, I would suggest the following:

You could use image.debug=true for additional information about the deployment, although in this case, you may only see some messages regarding MongoDB starting in the background.
Use diagnosticMode.enabled=true to deploy the pods without running a sleep infinity, then open a bash terminal and initialize them manually. For shardsvr you can run the default '/entrypoint/replicaset-entrypoint.sh', but for configsvr I would recommend initializing the app step by step and see if the.

It may be complicated to do, but it is all we can do without the resources to reproduce the issue.

Aug 22 '22 11:08 migruiz4

@migruiz4 Thanks for the troubleshooting steps. I will close this as this probably doesn’t have to do with the chart itself

Aug 22 '22 11:08 jdholtz

@migruiz4 I have some more information about this problem and would like to get your input on any possible fixes.

It seems to be only one cluster I have that it fails in. The cause is that the script does not log in to the primary replicaset which explains why it fails to create the root user.

I cannot reproduce this if I manually do the same steps the script is doing. The helm chart says it supports Kubernetes versions 1.19+, but maybe it supports only later versions than that (the cluster I am running is 1.22).

Sorry I don’t have much information as to what the cause for this is, but have you run into any issue like this before? I appreciate the help.

Aug 30 '22 11:08 jdholtz