AranggDB operator deployment cannot schedule pod deployment

[ekaley]

Deployment

Using bosh CFCR to deploy a 1 master 3 worker cluster on vmware vcenter. Bosh CFCR(KUBO) release of 0.32.0

Observed Behavior When using the local storage operator for arango deployment with this config

apiVersion: "storage.arangodb.com/v1alpha"
kind: "ArangoLocalStorage"
metadata:
  name: "arangodb-local-storage"
spec:
  storageClass:
    name: local-ssd
    isDefault: true
  localPath:
  - /var/vcap/data

and deploying a single arango server or a cluster of servers, the deployment pods never start.

We debugged this issue to be that the kubernetes.io/hostname label on the CFCR nodes have an IP address not the VM hostname. K8s docs allow that here: https://kubernetes.io/docs/concepts/architecture/nodes/#addresses

Attempted work around We used nodeSelector as documented here: https://github.com/arangodb/kube-arangodb/blob/master/docs/Manual/Deployment/Kubernetes/StorageResource.md#specnodeselector-nodeselector

to manually attach provisioned storage to nodes.

However we found that the field is not being respected and the operator is looking for the hostname of the VM to be in the label no matter what is specified.

See code reference here: https://github.com/arangodb/kube-arangodb/blob/fb9cd7968e4009dea5cacda388de506a964281bb/pkg/storage/pv_creator.go#L230

TEMP FIX

We can issue kubectl edit nodes and manually edit the kubernetes.io/hostname label to be the VM hostname from the CFCR deployment instead of the node IP and the operator is able to schedule the PV and pod placement.

Spoke with engineering and will further elaborate below.

[developeer]

@kvahed we will be upload the bosh configuration soon.

[developeer]

@kvahed attached is the bosh deployment scripts. Let me know if you need any additional information.

copy_bosh.zip