How to integrate Custom StorageClass with OBCluster Deployment

[qchenzi]

Description

Hello everybody, I am currently implementing the OceanBase database within a Kubernetes environment that employs a custom scheduler and StorageClass. Our deployment has specific requirements for PVC bindings, which are currently challenging to implement directly in the OBCluster configuration.

Requirements:

1. Pod Annotations: Our deployment strategy involves annotating pods with specific labels to meet the scheduling and storage policies. For example, in a typical StatefulSet, we can add the necessary annotations like this:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myobcluster
spec:
  template:
    metadata:
      annotations:
        volumeType: LocalStorage  # the necessary annotation
    spec:
      schedulerName: ${schedulerName} # the necessary schedulerName

However, I am unsure how to add similar annotations within an OBCluster deployment to ensure the pods meet our custom scheduler's requirements.

2. Volume Claim Templates: Our PVCs need to be annotated with specific labels and follow certain naming conventions to align with our storage infrastructure. Typically, this is handled as follows:

volumeClaimTemplates:
  - metadata:
      name: data      # the certain name
      annotations:
        operator.datamars.org/diskType: local_ssd   # the necessary annotation
    spec:
      storageClassName: ${storageClassName}
      containers:
      ...

Questions:

• How can we configure pod annotations specifically for OBCluster pods?

• What is the recommended approach to set annotations and naming for volumeClaimTemplates in an OBCluster environment?

This customization is crucial for integrating OceanBase with our existing Kubernetes infrastructure, which has customize scheduling and storage mechanisms.

I would appreciate any documentation, examples, or guidance on how to achieve these configurations.

Thank you for your support and looking forward to your suggestions.

Best regards

[powerfooI]

Hi @qchenzi! Glad to see you here!

1. For the question in title, you could set custom storageClass for OBCluster pods in spec of OBCluster like following, (OBCluster YAML parameters is available at doc Create a cluster)

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBCluster
metadata:
  name: test
  namespace: oceanbase
  annotations:
    "oceanbase.oceanbase.com/independent-pvc-lifecycle": "true" 
    # "oceanbase.oceanbase.com/mode": "standalone" or "service"
    # "oceanbase.oceanbase.com/single-pvc": "true" 
spec:
  clusterName: obcluster
  clusterId: 1
  serviceAccount: "default"
  # ...
  observer:
    image: oceanbase/oceanbase-cloud-native:4.2.1.6-106000012024042515
    resource:
      cpu: 2
      memory: 10Gi
    storage:
      dataStorage:
-        storageClass: local-path
+        storageClass: your-customized-sc
        size: 50Gi
      redoLogStorage:
-        storageClass: local-path
+        storageClass: your-customized-sc
        size: 50Gi
      logStorage:
-        storageClass: local-path
+        storageClass: your-customized-sc
        size: 20Gi
  monitor:
    image: oceanbase/obagent:4.2.0-100000062023080210
    resource:
      cpu: 1
      memory: 1Gi

2. We are regret that by now there is no method to add customized annotations and schedulerName for OBCluster pods. Maybe we could add a field named PodFields in OBCluster type within which there are some unusual fields of PodSpec in next release (2.2.3 or 2.3.0). And any PR is welcomed!

By the way, may I ask you why do you implement your own k8s scheduler? What will it benefit your cluster management? :) Look forward to your reply.

[qchenzi]

Hi @powerfooI,

Thank you for your detailed response and the example for setting a custom StorageClass in the OBCluster specification.

Regarding the inability to add custom annotations and scheduler names directly to OBCluster pods, I am very appreciative of the proposed solution to include a PodFields attribute in future releases. I am eager to contribute to the development of this feature, as it aligns perfectly with our operational needs. However, I am new to the process of contributing to this project and would appreciate some guidance on how to get started. Could you please provide me with some initial steps or direct me to any resources that could help me understand where and how I can be most effective in my contributions?

As for our custom Kubernetes scheduler, it is an enhancement based on modifications to the scheduler-plugins. This customized scheduler is vital for our deployment strategy because it necessitates specific annotations on both pods and PVCs, which carry certain labels for successful scheduling of PVCs. These annotations and labels are not just for operational alignment but are also crucial for managing and monitoring PVC usage effectively across our cluster. This level of customization helps us in precisely tracking resource allocation and utilization, which is essential for optimizing our overall cluster management.

Additionally, we have been using the enterprise version of OceanBase and have a deep collaboration with your team. We are now planning to integrate the ob-operator into our internal cloud platform to facilitate the containerized deployment of OceanBase. This step is aimed at enhancing our operational capabilities and further leveraging OceanBase’s robust features within our Kubernetes environment.

Looking forward to collaborating on enhancing the OBCluster capabilities. Thank you for considering the integration of these features.

[powerfooI]

It's quite nice to know you have intention to contribute!

There are several existing docs on our project gh-page that demonstrate architecture of ob-operator, how to develop and deploy ob-operator as well as OBCluster on your laptop.

• Architecture
• Deploy ob-operator locally
• Develop ob-operator locally

OBCluster, OBZone and OBServer are of a three-layer structure. Configurations like storage, resources and other pod fields are transmit from top to bottom, from OBCluster's specification to OBZone's and finally to OBServer's.

ob-operator adopts a state machine pattern which updates status of a resource and run a specific task flow to align the spec and status. You could find out configuration transmission in obxxx_task.go. e.g. an OBClsuter creates OBZones in its CreateOBZone task, an OBZone creates OBServers in its CreateOBServer task, an OBServer creates Pod and PVC in its CreateOBServerPod and CreateOBServerPVC task.

We also recommend you join our dingtalk group if you need more help :)

[qchenzi]

Hi @powerfooI,

Thank you so much for the detailed information and the pointers to the relevant documentation on the architecture and development process of the ob-operator.

I would also love to join the DingTalk group for more direct communication and assistance. Could you please provide me with the details or an invitation to join the group? My DingTalk ID is p9vs5u2.

[powerfooI]

Of course! You can join our user group by scanning the following QRCode. I will contact you in it.😁

[chris-sun-star]

https://github.com/oceanbase/ob-operator/pull/515

[chris-sun-star]

Thanks @qchenzi for your excellent work. Hope we can have more collaborations later on.