Unable to access ArangoDB when using Kubernetes secret for bootstrapping root password

[345paul]

Summary

To start, this is inconsistent in the way that it works. Sometimes it will work, other times, it doesn't even though the log output in the deployment operator looks ok. When it doesn't seem to set the root password correctly, the password isn't blank, but it also isn't the value in the Kubernetes secret. I deploy ArangoDB via Helm so in the case where I cannot login into Arango, if I simply delete the chart in Helm and reinstall it will work fine.

The operator logs always shows:

2019-07-11T17:19:04Z DBG ArangoDeployment updated component=operator name=arangodb-catalog operator-id=45kkv
2019-07-11T17:19:05Z INF Bootstrap deployment arangodb-catalog component=deployment deployment=arangodb-catalog operator-id=45kkv
2019-07-11T17:19:05Z DBG Bootstrapping user root, secret arangodb-catalog-credentials-root component=deployment deployment=arangodb-catalog operator-id=45kkv
2019-07-11T17:19:05Z INF Bootstrap completed for arangodb-catalog component=deployment deployment=arangodb-catalog operator-id=45kkv

In the cases where I cannot login (either through the web interface, or running Arangosh), the output in the logs remains the same, as in, it looked like it worked.

Scenario

My Helm chart for ArangoDB includes the use of an ExternalSecret resource (using the GoDaddy external secret syncer Kubernetes controller), which syncs an AWS Secrets Manager secret with a Kubernetes Secret resource.

When the Helm chart is deployed and the ExternalSecret is created, it takes a little bit of time for the external secret controller to create a new Kubernetes secret. When the operator goes to bootstrap the root password, the Kubernetes secret doesn't exist and according to the documentation, it creates a random password.

This doesn't always happen - sometimes I'll deploy my Helm chart and the root password is successfully set and I can access the web interface.

If I remove the Helm chart and re-deploy, the Kubernetes secret still exists and so when the Arango operator bootstraps the root password, it works fine.

I have used the Auto option and it creates a specificly named secret, but I don't get one of those secrets, I only have the one that has been created by the external secrets controller.

Ideal Outcome / Fix

The log output doesn't give any indication that it couldn't find the secret and therefore it's setting a random password. This is misleading because if I have explicitly said in the values for the Helm chart to use a particular secret, a better option would be to either back-off and retry waiting for the secret, or simply fail to say the secret doesn't exist. The net result is that the ArangoDB instance cannot be accessed and therefore has to be removed and re-deployed.

The issue appears to be here in the code: https://github.com/arangodb/kube-arangodb/blob/master/pkg/deployment/resources/secrets.go#L105

		if err := k8sutil.CreateTokenSecret(secrets, secretName, token, &owner); k8sutil.IsAlreadyExists(err) {
			// Secret added while we tried it also
			return nil
		} else if err != nil {
			// Failed to create secret
			return maskAny(err)
		}

It has already determined that the secret doesn't exist and it must create it, however, when it creates it, it might have already been created and so it silently ignores it. You end up in a situation where Arango has been set with a random password, but the secret has a completely different password.

I'm not aware of how the password gets set on ArangoDB and whereabouts in the process, but unless it re-reads the secret in this instance from Kubernetes, then it will use the random password but the Kubernetes secret won't have the same value.

Documentation

The documentation on the bootstrap option in the Helm chart for the Arango operator doesn't say in what format the secret in Kubernetes must be in. I had to look up in the code (https://github.com/arangodb/kube-arangodb/blob/master/pkg/util/k8sutil/secrets.go#L300 and https://github.com/arangodb/kube-arangodb/blob/master/pkg/util/constants/constants.go#L44) to find that it needs a both a username and a password property in the secret and the username must be root. Maybe this could be added to the documentation?

[345paul]

My mistake, I think the bootstrap code is actually here, but I can't figure out what would cause the behaviour I'm seeing.

https://github.com/arangodb/kube-arangodb/blob/master/pkg/deployment/bootstrap.go#L81