[Feature] Add primary / secondary labels to pods

[hedgieinsocks]

Please, consider labeling pods in the SemiSync Replication cluster as primary / secondary to allow k8s admins to use label selectors when listing pods, or when there is a need to create a custom service with a custom port (for a sidecar) that should target only either master or replica.

Alternatively, allow specifying custom ports in https://github.com/mariadb-operator/mariadb-operator/blob/main/docs/API_REFERENCE.md#servicetemplate

Environment details:

• Kubernetes version: [1.25]
• Kubernetes distribution: [Rancher]
• mariadb-operator version: [0.29.0]
• Install method: [helm]
• Install flavor: [recommended]

[mmontes11]

Hey there @hedgieinsocks !

Please, consider labeling pods in the SemiSync Replication cluster as primary / secondary to allow k8s admins to use label selectors when listing pods

I'm affraid that this will not be possible, we are currently using a StatefulSet, which has the same Pod template for both the primary and secondary Pods.

there is a need to create a custom service with a custom port (for a sidecar) that should target only either master or replica.

We already have custom services to address primary and secondary Pods. https://github.com/mariadb-operator/mariadb-operator/blob/main/docs/HA.md#kubernetes-services

custom port (for a sidecar)

This is not currently supported, but it could be relatively easy to achieve by:

• Adding a new servicePorts property to the MariaDB object:

apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
   servicePort:
     - name: custom-port
       port: 1234

type MariaDBSpec struct {
	// ServicePorts are additional ports to be added to the Services created by the operator.
	// +optional
	// +operator-sdk:csv:customresourcedefinitions:type=spec,xDescriptors={"urn:alm:descriptor:com.tectonic.ui:advanced"}
	ServicePorts []corev1.ServicePort `json:"servicePorts,omitempty"`
}

• Then, in the MariaDB controller, we will need to add the previous ports to the Ports option passed to the Service builder: https://github.com/mariadb-operator/mariadb-operator/blob/6171b061f944e10e29197fbdc7e4d1144b08352d/internal/controller/mariadb_controller.go#L512 https://github.com/mariadb-operator/mariadb-operator/blob/6171b061f944e10e29197fbdc7e4d1144b08352d/internal/controller/mariadb_controller.go#L566 https://github.com/mariadb-operator/mariadb-operator/blob/6171b061f944e10e29197fbdc7e4d1144b08352d/internal/controller/mariadb_controller.go#L590 https://github.com/mariadb-operator/mariadb-operator/blob/6171b061f944e10e29197fbdc7e4d1144b08352d/internal/controller/mariadb_controller.go#L618

Contributions are welcome!

[crabique]

@mmontes11 would it make sense to set the labels by the controller directly? Since it already watches pods and knows each pod's role, can't the labels be applied to all pods in an STS?

Other database operators also use StatefulSet, but the pod labels are continuously reconciled by the controller, e.g.: https://github.com/bitpoke/mysql-operator/blob/37c64109855691ab227130cf2d011effdd48690b/pkg/controller/mysqlcluster/internal/syncer/pod.go#L90-L112

In their case it's also used as targets for the service's label selectors, but the administrative side of it is also important: it's the most convenient place to look at/filter by when performing manual operations.

[mmontes11]

@crabique interesting, thanks for pointing this out.

I've manually added a label to the Pods, and it seems like the StatefulSet is not un-doing my changes, Good!

In that case, yes, we could add another reconciliation phase in the MariaDB controller, just after the StatefulSet phase: https://github.com/mariadb-operator/mariadb-operator/blob/6171b061f944e10e29197fbdc7e4d1144b08352d/internal/controller/mariadb_controller.go#L152

In this phase we would:

• List the available Pods
• Check which of them is the primary by looking at mariadb.Status.CurrentPrimaryPodIndex
• Patch each Pod with its corresponding k8s.mariadb.com/role labels (metadata defined here). Possible values: primary, replica

We are currently doing something very similar to reconcile the endpoints for the Secondary service: https://github.com/mariadb-operator/mariadb-operator/blob/6171b061f944e10e29197fbdc7e4d1144b08352d/pkg/controller/endpoints/controller.go#L89

If this new role labels work as expected, we could get rid of the previous code to build endpoints dynamically and use the k8s.mariadb.com/role in our Services. This is out of the scope of this issue, could be tackled separately.

Contributions welcome!