kubeblocks icon indicating copy to clipboard operation
kubeblocks copied to clipboard

[BUG] The MySQL 8.0.39 cluster fails to be created.

Open NovXxf opened this issue 2 months ago • 4 comments

Describe the bug The MySQL 8.0.39 cluster fails to be created.

To Reproduce Steps to reproduce the behavior:

  1. kbcli cluster create mysql cmdb-mysql --namespace devops --version='8.0.39' --storage=10 --replicas=2
  2. see errors

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information): Kubernetes: v1.26.15 KubeBlocks: 1.0.1 kbcli: 1.0.1

Additional context

2025-10-30T03:07:58Z    INFO    HTTPServer      Starting HTTP Server
2025-10-30T03:07:58Z    INFO    HTTPServer      API route path  {"method": "POST", "path": ["/v1.0/pause", "/v1.0/switchover", "/v1.0/leavemember", "/v1.0/rebuild", "/v1.0/resume", "/v1.0/start", "/v1.0/stop"]}
2025-10-30T03:07:58Z    INFO    HTTPServer      API route path  {"method": "GET", "path": ["/v1.0/datasync", "/v1.0/getrole"]}
2025-10-30T03:07:58Z    INFO    HTTPServer      http server     {"listen address": "0.0.0.0", "port": 3601}
2025-10-30T03:07:58Z    INFO    HA      HA starting
2025-10-30T03:07:58Z    INFO    pinger  Waiting for dns resolution to be ready
2025-10-30T03:07:58Z    INFO    pinger  dns resolution is ready {"dns": ""}
2025-10-30T03:07:58Z    INFO    DCS-K8S pod selector: app.kubernetes.io/instance=cmdb-mysql,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql
2025-10-30T03:07:58Z    INFO    DCS-K8S podlist: 2
2025-10-30T03:07:58Z    INFO    DCS-K8S Leader configmap is not found   {"configmap": "cmdb-mysql-mysql-leader"}
2025-10-30T03:07:58Z    DEBUG   HA      cluster info    {"cluster": "cmdb-mysql-mysql", "replicas": 2}
2025-10-30T03:07:58Z    INFO    Hypervisor      Starting Hypervisor
2025-10-30T03:07:58Z    INFO    Hypervisor      Start DB Service        {"command": "/usr/bin/bash -c mkdir -p /var/lib/mysql/{log,binlog,auditlog,temp}\nif [ -f /var/lib/mysql/plugin/audit_log.so ]; then\n  cp /var/lib/mysql/plugin/audit_log.so /usr/lib64/mysql/plugin/\nfi \nif [ -d /etc/pki/tls ]; then\n  mkdir -p /var/lib/mysql/tls/\n  cp -L /etc/pki/tls/*.pem /var/lib/mysql/tls/\n  chmod 600 /var/lib/mysql/tls/*\nfi\nchown -R mysql:root /var/lib/mysql\nSERVICE_ID=$((${POD_NAME##*-} + 1))\n\ndocker-entrypoint.sh mysqld --server-id $SERVICE_ID \\\n   --plugin-load-add=rpl_semi_sync_source=semisync_source.so \\\n   --plugin-load-add=rpl_semi_sync_replica=semisync_replica.so \\\n   --plugin-load-add=audit_log=audit_log.so \\\n   --log-bin=/var/lib/mysql/binlog/${POD_NAME}-bin \\\n   --skip-slave-start=ON\n"}
2025-10-30T03:07:58Z    INFO    HA      check if DB Service is running
2025-10-30T03:07:58Z    INFO    Hypervisor      Starting watcher on dbService
2025-10-30T03:07:58Z    INFO    MySQL   DB is not ready and try to reconnect    {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"}
2025-10-30T03:07:58Z    INFO    MySQL   switch to root db
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
2025-10-30T03:07:58Z    INFO    DCS-K8S Leader configmap is not found   {"configmap": "cmdb-mysql-mysql-leader"}
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == <jemalloc>: Unsupported system page size
== DB == 2025-10-30 03:07:58+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.39-1.el9 started.
== DB ERR == <jemalloc>: Unsupported system page size
== DB ERR == 2025-10-30 03:07:58+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
== DB ERR ==    command was: mysqld --server-id 2 --plugin-load-add=rpl_semi_sync_source=semisync_source.so --plugin-load-add=rpl_semi_sync_replica=semisync_replica.so --plugin-load-add=audit_log=audit_log.so --log-bin=/var/lib/mysql/binlog/cmdb-mysql-mysql-1-bin --skip-slave-start=ON --verbose --help --log-bin-index=/tmp/tmp.XUPNkFPJIh
== DB ERR ==    <jemalloc>: Unsupported system page size
== DB ERR == terminate called after throwing an instance of 'std::bad_alloc'
== DB ERR ==   what():  std::bad_alloc
2025-10-30T03:07:58Z    INFO    Hypervisor      dbService existing      {"state": "exit status 1"}
2025-10-30T03:07:58Z    INFO    Hypervisor      db startup failed, and will not be watched, Existing... {"daemon": "/usr/bin/bash -c mkdir -p /var/lib/mysql/{log,binlog,auditlog,temp}\nif [ -f /var/lib/mysql/plugin/audit_log.so ]; then\n  cp /var/lib/mysql/plugin/audit_log.so /usr/lib64/mysql/plugin/\nfi \nif [ -d /etc/pki/tls ]; then\n  mkdir -p /var/lib/mysql/tls/\n  cp -L /etc/pki/tls/*.pem /var/lib/mysql/tls/\n  chmod 600 /var/lib/mysql/tls/*\nfi\nchown -R mysql:root /var/lib/mysql\nSERVICE_ID=$((${POD_NAME##*-} + 1))\n\ndocker-entrypoint.sh mysqld --server-id $SERVICE_ID \\\n   --plugin-load-add=rpl_semi_sync_source=semisync_source.so \\\n   --plugin-load-add=rpl_semi_sync_replica=semisync_replica.so \\\n   --plugin-load-add=audit_log=audit_log.so \\\n   --log-bin=/var/lib/mysql/binlog/${POD_NAME}-bin \\\n   --skip-slave-start=ON\n"}

NovXxf avatar Oct 30 '25 03:10 NovXxf

instanceset.yaml here

apiVersion: workloads.kubeblocks.io/v1
kind: InstanceSet
metadata:
  annotations:
    app.kubernetes.io/component: mysql-8.0-1.0.1
    apps.kubeblocks.io/replicas-status: '{"replicas":2,"status":[{"name":"cmdb-mysql-mysql-0","generation":"2","creationTimestamp":"0001-01-01T00:00:00Z","provisioned":true},{"name":"cmdb-mysql-mysql-1","generation":"2","creationTimestamp":"0001-01-01T00:00:00Z","provisioned":true}]}'
    apps.kubeblocks.io/service-version: 8.0.39
    kubeblocks.io/crd-api-version: workloads.kubeblocks.io/v1
    kubeblocks.io/generation: "2"
  creationTimestamp: "2025-10-30T02:46:28Z"
  finalizers:
  - component.kubeblocks.io/finalizer
  - instanceset.workloads.kubeblocks.io/finalizer
  generation: 1
  labels:
    app.kubernetes.io/instance: cmdb-mysql
    app.kubernetes.io/managed-by: kubeblocks
    apps.kubeblocks.io/component-name: mysql
  name: cmdb-mysql-mysql
  namespace: devops
  ownerReferences:
  - apiVersion: apps.kubeblocks.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Component
    name: cmdb-mysql-mysql
    uid: f581b320-b987-47c4-9950-b2db925b3723
  resourceVersion: "16090976"
  uid: 497fbea2-dc8f-4b58-87a5-c8f51549e138
spec:
  defaultTemplateOrdinals: {}
  memberUpdateStrategy: BestEffortParallel
  membershipReconfiguration:
    switchover:
      exec:
        command:
        - /bin/sh
        - -c
        - |2

          if [ "$KB_SWITCHOVER_ROLE" != "primary" ]; then
              echo "switchover not triggered for primary, nothing to do, exit 0."
              exit 0
          fi

          /tools/syncerctl switchover --primary "$KB_SWITCHOVER_CURRENT_NAME" ${KB_SWITCHOVER_CANDIDATE_NAME:+--candidate "$KB_SWITCHOVER_CANDIDATE_NAME"}
      timeoutSeconds: 0
  minReadySeconds: 0
  parallelPodManagementConcurrency: 100%
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
    whenScaled: Delete
  podManagementPolicy: OrderedReady
  podUpdatePolicy: PreferInPlace
  replicas: 2
  roles:
  - name: primary
    participatesInQuorum: false
    updatePriority: 2
  - name: secondary
    participatesInQuorum: false
    updatePriority: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: cmdb-mysql
      app.kubernetes.io/managed-by: kubeblocks
      apps.kubeblocks.io/component-name: mysql
  template:
    metadata:
      labels:
        app.kubernetes.io/component: mysql-8.0-1.0.1
        app.kubernetes.io/instance: cmdb-mysql
        app.kubernetes.io/managed-by: kubeblocks
        apps.kubeblocks.io/component-name: mysql
        apps.kubeblocks.io/release-phase: stable
        apps.kubeblocks.io/service-version: 8.0.39
    spec:
      containers:
      - command:
        - syncer
        - --port
        - "3601"
        - --
        - bash
        - -c
        - "mkdir -p /var/lib/mysql/{log,binlog,auditlog,temp}\nif [ -f /var/lib/mysql/plugin/audit_log.so
          ]; then\n  cp /var/lib/mysql/plugin/audit_log.so /usr/lib64/mysql/plugin/\nfi
          \nif [ -d /etc/pki/tls ]; then\n  mkdir -p /var/lib/mysql/tls/\n  cp -L
          /etc/pki/tls/*.pem /var/lib/mysql/tls/\n  chmod 600 /var/lib/mysql/tls/*\nfi\nchown
          -R mysql:root /var/lib/mysql\nSERVICE_ID=$((${POD_NAME##*-} + 1))\n\ndocker-entrypoint.sh
          mysqld --server-id $SERVICE_ID \\\n   --plugin-load-add=rpl_semi_sync_source=semisync_source.so
          \\\n   --plugin-load-add=rpl_semi_sync_replica=semisync_replica.so \\\n
          \  --plugin-load-add=audit_log=audit_log.so \\\n   --log-bin=/var/lib/mysql/binlog/${POD_NAME}-bin
          \\\n   --skip-slave-start=ON\n"
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: PATH
          value: /tools/xtrabackup/bin:/tools/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        - name: LD_PRELOAD
          value: /tools/lib/libjemalloc.so.2
        - name: KB_SERVICE_CHARACTER_TYPE
          value: mysql
        - name: MYSQL_INITDB_SKIP_TZINFO
          value: "1"
        - name: MYSQL_ROOT_HOST
          value: '%'
        - name: SERVICE_PORT
          value: "3306"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/mysql:8.0.39
        imagePullPolicy: IfNotPresent
        name: mysql
        ports:
        - containerPort: 3306
          name: mysql
          protocol: TCP
        - containerPort: 3601
          name: ha
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: data
        - mountPath: /etc/mysql/conf.d
          name: mysql-config
        - mountPath: /scripts
          name: scripts
        - mountPath: /tools
          name: tools
      - command:
        - bash
        - -c
        - |
          mysqld_exporter --mysqld.username=${MYSQLD_EXPORTER_USER} --web.listen-address=:${EXPORTER_WEB_PORT} --log.level=info
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQLD_EXPORTER_USER
          value: $(MYSQL_ROOT_USER)
        - name: MYSQLD_EXPORTER_PASSWORD
          value: $(MYSQL_ROOT_PASSWORD)
        - name: EXPORTER_WEB_PORT
          value: "9104"
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/mysqld-exporter:0.15.1
        imagePullPolicy: IfNotPresent
        name: mysql-exporter
        ports:
        - containerPort: 9104
          name: http-metrics
          protocol: TCP
        resources:
          limits:
            cpu: "0"
            memory: "0"
        volumeMounts:
        - mountPath: /scripts
          name: scripts
      - args:
        - --port
        - "3501"
        - --streaming-port
        - "3502"
        command:
        - /kubeblocks/kbagent
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: KB_AGENT_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: KB_AGENT_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: KB_AGENT_POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: KB_AGENT_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: KB_AGENT_ACTION
          value: '[{"name":"switchover","exec":{"command":["/bin/sh","-c","\nif [
            \"$KB_SWITCHOVER_ROLE\" != \"primary\" ]; then\n    echo \"switchover
            not triggered for primary, nothing to do, exit 0.\"\n    exit 0\nfi\n\n/tools/syncerctl
            switchover --primary \"$KB_SWITCHOVER_CURRENT_NAME\" ${KB_SWITCHOVER_CANDIDATE_NAME:+--candidate
            \"$KB_SWITCHOVER_CANDIDATE_NAME\"}\n"]}},{"name":"accountProvision","exec":{"command":["bash","-c","set
            -ex\nALL_DB=''*.*''\neval statement=\\\"${KB_ACCOUNT_STATEMENT}\\\"\nmysql
            -u${MYSQL_ROOT_USER} -p${MYSQL_ROOT_PASSWORD} -P3306 -h127.0.0.1 -e \"${statement}\"\n"]}},{"name":"roleProbe","exec":{"command":["/tools/syncerctl","getrole"]},"timeoutSeconds":1}]'
        - name: KB_AGENT_PROBE
          value: '[{"instance":"cmdb-mysql-mysql","action":"roleProbe","periodSeconds":1}]'
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/mysql:8.0.39
        imagePullPolicy: IfNotPresent
        name: kbagent
        ports:
        - containerPort: 3501
          name: http
          protocol: TCP
        - containerPort: 3502
          name: streaming
          protocol: TCP
        resources:
          limits:
            cpu: "0"
            memory: "0"
        securityContext:
          runAsGroup: 1000
        startupProbe:
          tcpSocket:
            port: 3501
        volumeMounts:
        - mountPath: /kubeblocks
          name: kubeblocks
        - mountPath: /var/lib/mysql
          name: data
        - mountPath: /etc/mysql/conf.d
          name: mysql-config
        - mountPath: /scripts
          name: scripts
        - mountPath: /tools
          name: tools
      - args:
        - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
        - /bin/reloader
        - --log-level
        - info
        - --operator-update-enable
        - --tcp
        - "9901"
        - --config
        - /opt/config-manager/config-manager.yaml
        command:
        - env
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: CONFIG_MANAGER_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: TOOLS_PATH
          value: /opt/kb-tools/reload/mysql-replication-config:/opt/config-manager
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:1.0.1
        imagePullPolicy: IfNotPresent
        name: config-manager
        ports:
        - containerPort: 9901
          name: config-manager
          protocol: TCP
        resources:
          limits:
            cpu: "0"
            memory: "0"
        volumeMounts:
        - mountPath: /etc/mysql/conf.d
          name: mysql-config
        - mountPath: /opt/kb-tools/reload/mysql-replication-config
          name: cm-script-mysql-replication-config
        - mountPath: /opt/config-manager
          name: config-manager-config
      initContainers:
      - command:
        - bash
        - -c
        - |
          mkdir -p /var/lib/mysql/{log,binlog,auditlog}
          cp /usr/lib/mysql/plugin/ /var/lib/mysql/plugin -r
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/mysql_audit_log:8.0.33
        imagePullPolicy: IfNotPresent
        name: init-data
        resources:
          limits:
            cpu: "0"
            memory: "0"
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: data
      - command:
        - cp
        - -r
        - /xtrabackup
        - /tools/xtrabackup
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/xtrabackup:8.0
        imagePullPolicy: IfNotPresent
        name: init-xtrabackup
        resources:
          limits:
            cpu: "0"
            memory: "0"
        volumeMounts:
        - mountPath: /tools
          name: tools
      - command:
        - cp
        - -r
        - /jemalloc/lib/
        - /tools/lib
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/jemalloc:5.3.0
        imagePullPolicy: IfNotPresent
        name: init-jemalloc
        resources:
          limits:
            cpu: "0"
            memory: "0"
        volumeMounts:
        - mountPath: /tools
          name: tools
      - command:
        - cp
        - -r
        - /bin/syncer
        - /bin/syncerctl
        - /tools/
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/syncer:0.5.0
        imagePullPolicy: IfNotPresent
        name: init-syncer
        resources:
          limits:
            cpu: "0"
            memory: "0"
        volumeMounts:
        - mountPath: /tools
          name: tools
      - command:
        - cp
        - -r
        - /bin/kbagent
        - /kubeblocks/
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:1.0.1
        imagePullPolicy: IfNotPresent
        name: init-kbagent
        resources:
          limits:
            cpu: "0"
            memory: "0"
        volumeMounts:
        - mountPath: /kubeblocks
          name: kubeblocks
      - args:
        - --server=false
        command:
        - /kubeblocks/kbagent
        env:
        - name: MYSQL_ROOT_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-root
        - name: MYSQL_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbadmin
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: cmdb-mysql-mysql-account-kbreplicator
        - name: KB_AGENT_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: KB_AGENT_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: KB_AGENT_POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: KB_AGENT_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: KB_AGENT_ACTION
          value: '[{"name":"switchover","exec":{"command":["/bin/sh","-c","\nif [
            \"$KB_SWITCHOVER_ROLE\" != \"primary\" ]; then\n    echo \"switchover
            not triggered for primary, nothing to do, exit 0.\"\n    exit 0\nfi\n\n/tools/syncerctl
            switchover --primary \"$KB_SWITCHOVER_CURRENT_NAME\" ${KB_SWITCHOVER_CANDIDATE_NAME:+--candidate
            \"$KB_SWITCHOVER_CANDIDATE_NAME\"}\n"]}},{"name":"accountProvision","exec":{"command":["bash","-c","set
            -ex\nALL_DB=''*.*''\neval statement=\\\"${KB_ACCOUNT_STATEMENT}\\\"\nmysql
            -u${MYSQL_ROOT_USER} -p${MYSQL_ROOT_PASSWORD} -P3306 -h127.0.0.1 -e \"${statement}\"\n"]}},{"name":"roleProbe","exec":{"command":["/tools/syncerctl","getrole"]},"timeoutSeconds":1}]'
        - name: KB_AGENT_PROBE
          value: '[{"instance":"cmdb-mysql-mysql","action":"roleProbe","periodSeconds":1}]'
        envFrom:
        - configMapRef:
            name: cmdb-mysql-mysql-env
            optional: false
        image: docker.1ms.run/apecloud/mysql:8.0.39
        imagePullPolicy: IfNotPresent
        name: kbagent-worker
        resources:
          limits:
            cpu: "0"
            memory: "0"
        securityContext:
          runAsGroup: 1000
        volumeMounts:
        - mountPath: /kubeblocks
          name: kubeblocks
        - mountPath: /var/lib/mysql
          name: data
        - mountPath: /etc/mysql/conf.d
          name: mysql-config
        - mountPath: /scripts
          name: scripts
        - mountPath: /tools
          name: tools
      serviceAccountName: kb-mysql-8.0-1.0.1
      volumes:
      - emptyDir: {}
        name: data
      - emptyDir: {}
        name: tools
      - emptyDir: {}
        name: kubeblocks
      - configMap:
          defaultMode: 292
          name: cmdb-mysql-mysql-mysql-replication-config
        name: mysql-config
      - configMap:
          defaultMode: 365
          name: cmdb-mysql-mysql-mysql-scripts
        name: scripts
      - configMap:
          defaultMode: 493
          name: sidecar-mysql8.0-reload-script-cmdb-mysql
        name: cm-script-mysql-replication-config
      - configMap:
          defaultMode: 493
          name: sidecar-cmdb-mysql-mysql-config-manager-config
        name: config-manager-config
  templateVars:
    CLUSTER_COMPONENT_NAME: cmdb-mysql-mysql
    CLUSTER_NAME: cmdb-mysql
    CLUSTER_NAMESPACE: devops
    CLUSTER_UUID: 2ef3f6fc-806f-4aa6-8279-eb71c1f2f399
    COMPONENT_NAME: mysql
  volumeClaimTemplates:
  - metadata:
      labels:
        apps.kubeblocks.io/vct-name: data
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
    status: {}
status:
  availableReplicas: 0
  conditions:
  - lastTransitionTime: "2025-10-30T02:46:42Z"
    message: '["cmdb-mysql-mysql-0","cmdb-mysql-mysql-1"]'
    observedGeneration: 1
    reason: NotReady
    status: "False"
    type: InstanceReady
  - lastTransitionTime: "2025-10-30T02:46:42Z"
    message: '[]'
    observedGeneration: 1
    reason: NotAvailable
    status: "False"
    type: InstanceAvailable
  - lastTransitionTime: "2025-10-30T03:08:16Z"
    message: '["cmdb-mysql-mysql-1"]'
    observedGeneration: 1
    reason: InstanceFailure
    status: "True"
    type: InstanceFailure
  currentReplicas: 2
  currentRevision: 5885f78588
  currentRevisions:
    cmdb-mysql-mysql-0: 5885f78588
    cmdb-mysql-mysql-1: 5885f78588
  initReplicas: 0
  instanceStatus:
  - podName: cmdb-mysql-mysql-0
  - podName: cmdb-mysql-mysql-1
  observedGeneration: 1
  replicas: 2
  updateRevision: 5885f78588
  updateRevisions:
    cmdb-mysql-mysql-0: 5885f78588
    cmdb-mysql-mysql-1: 5885f78588
  updatedReplicas: 2

NovXxf avatar Oct 30 '25 03:10 NovXxf

For some ARM environments, please remove LD_PRELOAD=/tools/lib/libjemalloc.so.2.

  1. edit cmpd kubectl edit cmpd mysql-8.0

  2. remove LD_PRELOAD export LD_PRELOAD=/tools/lib/libjemalloc.so.2

  3. skip-immutable-check kubectl annotate cmpd mysql-8.0 apps.kubeblocks.io/skip-immutable-check=true

JashBook avatar Oct 30 '25 03:10 JashBook

Thank you very much, and we also need to do this:

kubectl annotate cmpd<CMPD_NAME> apps.kubeblocks.io/skip-immutable-check=true

NovXxf avatar Oct 30 '25 04:10 NovXxf

  • jemalloc issue: https://github.com/jemalloc/jemalloc/issues/2639
  • facebook jemalloc PR: https://github.com/facebook/jemalloc/pull/34

@xuriwuyun maybe we can take the same solution from fackbook.

shanshanying avatar Nov 03 '25 11:11 shanshanying

This issue has been marked as stale because it has been open for 30 days with no activity

github-actions[bot] avatar Dec 08 '25 00:12 github-actions[bot]