Issue when upgrading from Kubeflow 1.3 to 1.5

[pablofiumara]

/kind bug

What steps did you take and what happened: I upgraded from Kubeflow v1.3 to Kubeflow v.15

What did you expect to happen:

I expected katib-db-manager and katib-mysql pods to be running. Instead their status is CrashLoopBackOff

Anything else you would like to add:

If I execute

kubectl logs katib-db-manager -n kubeflow I get

E0823 23:47:01.897501       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:06.889384       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:11.881359       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:16.873428       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:21.929521       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:26.921382       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:31.913463       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:36.905402       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:41.897387       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:46.889461       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:51.881540       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
E0823 23:47:56.873399       1 mysql.go:78] Ping to Katib db failed: dial tcp oneIp:3306: connect: connection refused
F0823 23:47:56.873464       1 main.go:99] Failed to open db connection: DB open failed: Timeout waiting for DB conn successfully opened.

If I execute

kubectl describe pod katib-mysql -n kubeflow I get

  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Normal   Scheduled               33m                default-scheduler        Successfully assigned kubeflow/katib-mysql to gke-kubeflowNameCluster-default-pool
  Normal   SuccessfulAttachVolume  33m                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-aoneId"
  Normal   Pulled                  32m (x4 over 33m)  kubelet                  Container image "mysql:8.0.26" already present on machine
  Normal   Created                 32m (x4 over 33m)  kubelet                  Created container katib-mysql
  Normal   Started                 32m (x4 over 33m)  kubelet                  Started container katib-mysql
  Warning  Unhealthy               32m (x3 over 33m)  kubelet                  Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
  Warning  BackOff  3m10s (x158 over 33m)  kubelet  Back-off restarting failed container

Environment:

• Katib version (check the Katib controller image version): v0.13.0
• Kubernetes version: (kubectl version): 1.21.1

---

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

[anencore94]

The reason why katib-db-manager keep being CrashLoopBackOff is because katib-mysql is not ready. However, the root cause of why your katib-mysql is being CrashLoopBackoff is hard to find with your answer. Could you also upload the kubectl get deploy katib-mysql -n kubeflow -o yaml and the k8s secret file if you use it to set MYSQL_ROOT_PASSWORD or other values for katib-mysql ? (If the password is private, you may shouldn't upload this value)

[pablofiumara]

@anencore94 Thank you for your answer

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "31"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"katib.kubeflow.org/component":"mysql"},"name":"katib-mysql","namespace":"kubeflow"},"spec":{"replicas":1,"selector":{"matchLabels":{"katib.kubeflow.org/component":"mysql"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"katib.kubeflow.org/component":"mysql"}},"spec":{"containers":[{"args":["--datadir","/var/lib/mysql/datadir"],"env":[{"name":"MYSQL_ROOT_PASSWORD","valueFrom":{"secretKeyRef":{"key":"MYSQL_ROOT_PASSWORD","name":"katib-mysql-secrets"}}},{"name":"MYSQL_ALLOW_EMPTY_PASSWORD","value":"true"},{"name":"MYSQL_DATABASE","value":"katib"}],"image":"mysql:8.0.26","livenessProbe":{"exec":{"command":["/bin/bash","-c","mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"]},"failureThreshold":10,"periodSeconds":2},"name":"katib-mysql","ports":[{"containerPort":3306,"name":"dbapi"}],"readinessProbe":{"exec":{"command":["/bin/bash","-c","mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'"]},"failureThreshold":10,"periodSeconds":2},"startupProbe":{"exec":{"command":["/bin/bash","-c","mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"]},"failureThreshold":60,"periodSeconds":15},"volumeMounts":[{"mountPath":"/var/lib/mysql","name":"katib-mysql"}]}],"volumes":[{"name":"katib-mysql","persistentVolumeClaim":{"claimName":"katib-mysql"}}]}}}}
  creationTimestamp: "2022-08-23T19:24:07Z"
  generation: 31
  labels:
    katib.kubeflow.org/component: mysql
  name: katib-mysql
  namespace: kubeflow
  resourceVersion: "675169595"
  uid: 01708f07-335e-43b7-ac67-518089728d86
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      katib.kubeflow.org/component: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-08-24T22:04:58-03:00"
        sidecar.istio.io/inject: "false"
      creationTimestamp: null
      labels:
        katib.kubeflow.org/component: mysql
    spec:
      containers:
      - args:
        - --datadir
        - /var/lib/mysql/datadir
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: MYSQL_ROOT_PASSWORD
              name: katib-mysql-secrets
        - name: MYSQL_ALLOW_EMPTY_PASSWORD
          value: "true"
        - name: MYSQL_DATABASE
          value: katib
        image: mysql:8.0.26
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}
          failureThreshold: 10
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 1
        name: katib-mysql
        ports:
        - containerPort: 3306
          name: dbapi
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT
              1'
          failureThreshold: 10
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        startupProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}
          failureThreshold: 60
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: katib-mysql
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: katib-mysql
        persistentVolumeClaim:
          claimName: katib-mysql
status:
  conditions:
  - lastTransitionTime: "2022-08-23T19:24:07Z"
    lastUpdateTime: "2022-08-23T19:24:07Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2022-08-25T01:32:14Z"
    lastUpdateTime: "2022-08-25T01:32:14Z"
    message: ReplicaSet "katib-mysql-574ffff889" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 31
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

v1_secret_katib-mysql-secrets.yaml

apiVersion: v1
data:
  MYSQL_ROOT_PASSWORD: onePassword
kind: Secret
metadata:
  name: katib-mysql-secrets
  namespace: kubeflow
type: Opaque

[anencore94]

For my k8s environments, with your secret, deployment and another simple pvc yaml works well. (except changing secret's value to some other value encoded by base64)

To make sure, does your onePassword means that it is just a hidden password, but you can see it is base64-encoded ? Also, would you mind to tell your k8s cluster's hardware spec ?

[midhun1998]

@anencore94 Thank you for your answer

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "31"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"katib.kubeflow.org/component":"mysql"},"name":"katib-mysql","namespace":"kubeflow"},"spec":{"replicas":1,"selector":{"matchLabels":{"katib.kubeflow.org/component":"mysql"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"katib.kubeflow.org/component":"mysql"}},"spec":{"containers":[{"args":["--datadir","/var/lib/mysql/datadir"],"env":[{"name":"MYSQL_ROOT_PASSWORD","valueFrom":{"secretKeyRef":{"key":"MYSQL_ROOT_PASSWORD","name":"katib-mysql-secrets"}}},{"name":"MYSQL_ALLOW_EMPTY_PASSWORD","value":"true"},{"name":"MYSQL_DATABASE","value":"katib"}],"image":"mysql:8.0.26","livenessProbe":{"exec":{"command":["/bin/bash","-c","mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"]},"failureThreshold":10,"periodSeconds":2},"name":"katib-mysql","ports":[{"containerPort":3306,"name":"dbapi"}],"readinessProbe":{"exec":{"command":["/bin/bash","-c","mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'"]},"failureThreshold":10,"periodSeconds":2},"startupProbe":{"exec":{"command":["/bin/bash","-c","mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"]},"failureThreshold":60,"periodSeconds":15},"volumeMounts":[{"mountPath":"/var/lib/mysql","name":"katib-mysql"}]}],"volumes":[{"name":"katib-mysql","persistentVolumeClaim":{"claimName":"katib-mysql"}}]}}}}
  creationTimestamp: "2022-08-23T19:24:07Z"
  generation: 31
  labels:
    katib.kubeflow.org/component: mysql
  name: katib-mysql
  namespace: kubeflow
  resourceVersion: "675169595"
  uid: 01708f07-335e-43b7-ac67-518089728d86
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      katib.kubeflow.org/component: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-08-24T22:04:58-03:00"
        sidecar.istio.io/inject: "false"
      creationTimestamp: null
      labels:
        katib.kubeflow.org/component: mysql
    spec:
      containers:
      - args:
        - --datadir
        - /var/lib/mysql/datadir
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: MYSQL_ROOT_PASSWORD
              name: katib-mysql-secrets
        - name: MYSQL_ALLOW_EMPTY_PASSWORD
          value: "true"
        - name: MYSQL_DATABASE
          value: katib
        image: mysql:8.0.26
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}
          failureThreshold: 10
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 1
        name: katib-mysql
        ports:
        - containerPort: 3306
          name: dbapi
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT
              1'
          failureThreshold: 10
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        startupProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}
          failureThreshold: 60
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: katib-mysql
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: katib-mysql
        persistentVolumeClaim:
          claimName: katib-mysql
status:
  conditions:
  - lastTransitionTime: "2022-08-23T19:24:07Z"
    lastUpdateTime: "2022-08-23T19:24:07Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2022-08-25T01:32:14Z"
    lastUpdateTime: "2022-08-25T01:32:14Z"
    message: ReplicaSet "katib-mysql-574ffff889" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 31
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

v1_secret_katib-mysql-secrets.yaml

apiVersion: v1
data:
  MYSQL_ROOT_PASSWORD: onePassword
kind: Secret
metadata:
  name: katib-mysql-secrets
  namespace: kubeflow
type: Opaque

@pablofiumara Is the katib-sql pod up and running? Looking at the yaml i felt like the DB isn't up and running. I might be wrong but you could clarify this too for @anencore94 to help debug. 🙂

[pablofiumara]

For my k8s environments, with your secret, deployment and another simple pvc yaml works well. (except changing secret's value to some other value encoded by base64)

To make sure, does your onePassword means that it is just a hidden password, but you can see it is base64-encoded ? Also, would you mind to tell your k8s cluster's hardware spec ?

Thanks for your answer. onePassword is base64-encoded

My k8s cluster's hardware spec is: 8vCPUs, 32GiB of Memory

[pablofiumara]

@anencore94 Thank you for your answer

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "31"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"katib.kubeflow.org/component":"mysql"},"name":"katib-mysql","namespace":"kubeflow"},"spec":{"replicas":1,"selector":{"matchLabels":{"katib.kubeflow.org/component":"mysql"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"katib.kubeflow.org/component":"mysql"}},"spec":{"containers":[{"args":["--datadir","/var/lib/mysql/datadir"],"env":[{"name":"MYSQL_ROOT_PASSWORD","valueFrom":{"secretKeyRef":{"key":"MYSQL_ROOT_PASSWORD","name":"katib-mysql-secrets"}}},{"name":"MYSQL_ALLOW_EMPTY_PASSWORD","value":"true"},{"name":"MYSQL_DATABASE","value":"katib"}],"image":"mysql:8.0.26","livenessProbe":{"exec":{"command":["/bin/bash","-c","mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"]},"failureThreshold":10,"periodSeconds":2},"name":"katib-mysql","ports":[{"containerPort":3306,"name":"dbapi"}],"readinessProbe":{"exec":{"command":["/bin/bash","-c","mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'"]},"failureThreshold":10,"periodSeconds":2},"startupProbe":{"exec":{"command":["/bin/bash","-c","mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"]},"failureThreshold":60,"periodSeconds":15},"volumeMounts":[{"mountPath":"/var/lib/mysql","name":"katib-mysql"}]}],"volumes":[{"name":"katib-mysql","persistentVolumeClaim":{"claimName":"katib-mysql"}}]}}}}
  creationTimestamp: "2022-08-23T19:24:07Z"
  generation: 31
  labels:
    katib.kubeflow.org/component: mysql
  name: katib-mysql
  namespace: kubeflow
  resourceVersion: "675169595"
  uid: 01708f07-335e-43b7-ac67-518089728d86
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      katib.kubeflow.org/component: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-08-24T22:04:58-03:00"
        sidecar.istio.io/inject: "false"
      creationTimestamp: null
      labels:
        katib.kubeflow.org/component: mysql
    spec:
      containers:
      - args:
        - --datadir
        - /var/lib/mysql/datadir
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: MYSQL_ROOT_PASSWORD
              name: katib-mysql-secrets
        - name: MYSQL_ALLOW_EMPTY_PASSWORD
          value: "true"
        - name: MYSQL_DATABASE
          value: katib
        image: mysql:8.0.26
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}
          failureThreshold: 10
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 1
        name: katib-mysql
        ports:
        - containerPort: 3306
          name: dbapi
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT
              1'
          failureThreshold: 10
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        startupProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}
          failureThreshold: 60
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: katib-mysql
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: katib-mysql
        persistentVolumeClaim:
          claimName: katib-mysql
status:
  conditions:
  - lastTransitionTime: "2022-08-23T19:24:07Z"
    lastUpdateTime: "2022-08-23T19:24:07Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2022-08-25T01:32:14Z"
    lastUpdateTime: "2022-08-25T01:32:14Z"
    message: ReplicaSet "katib-mysql-574ffff889" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 31
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

v1_secret_katib-mysql-secrets.yaml

apiVersion: v1
data:
  MYSQL_ROOT_PASSWORD: onePassword
kind: Secret
metadata:
  name: katib-mysql-secrets
  namespace: kubeflow
type: Opaque

@pablofiumara Is the katib-sql pod up and running? Looking at the yaml i felt like the DB isn't up and running. I might be wrong but you could clarify this too for @anencore94 to help debug. slightly_smiling_face

@midhun1998 Thank you for your answer. Pod is not up and running, it crashes

[johnugeorge]

@pablofiumara Can you share pod logs?

[pablofiumara]

@johnugeorge Thanks for your answer. Here they are:

If I execute

kubectl logs katib-mysql -n kubeflow

I get

2022-08-29 18:15:11+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.26-1debian10 started.
2022-08-29 18:15:12+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-08-29 18:15:12+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.26-1debian10 started.
2022-08-29T18:15:12.483740Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.26) starting as process 1
2022-08-29T18:15:12.498279Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-29T18:15:12.985928Z 1 [ERROR] [MY-013171] [InnoDB] Cannot boot server version 80026 on data directory built by version 80030. Downgrade is not supported
mysqld: Can't open file: 'mysql.ibd' (errno: 0 - )
2022-08-29T18:15:17.956257Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-08-29T18:15:17.957151Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2022-08-29T18:15:17.957636Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-08-29T18:15:17.958652Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.26)  MySQL Community Server - GPL.

[johnugeorge]

This issue is happening because of the mysql version has changed. In 1.3, version used was mysql:8 and 1.5, it is pinned to mysql:8.0.26

I haven't tried this. But, can you try keeping the same image version as the previous? 8.0.30

[pablofiumara]

@johnugeorge Thanks again.

If I change it to 8.0.30, I get the following messages from the logs

2022-08-29 18:54:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
2022-08-29 18:54:19+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-08-29 18:54:19+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
ln: failed to create symbolic link '/var/lib/mysql/mysql.sock': Permission denied
2022-08-29T18:54:19.815942Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2022-08-29T18:54:19.827978Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 1
2022-08-29T18:54:19.905531Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-29T18:54:20.511174Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 8.0.26, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2022-08-29T18:54:20.511404Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2022-08-29T18:54:20.704305Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-08-29T18:54:20.704641Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2022-08-29T18:54:20.704677Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-08-29T18:54:20.705483Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30)  MySQL Community Server - GPL.

[midhun1998]

@johnugeorge Thanks again.

If I change it to 8.0.30, I get the following messages from the logs

2022-08-29 18:54:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
2022-08-29 18:54:19+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-08-29 18:54:19+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
ln: failed to create symbolic link '/var/lib/mysql/mysql.sock': Permission denied
2022-08-29T18:54:19.815942Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2022-08-29T18:54:19.827978Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 1
2022-08-29T18:54:19.905531Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-29T18:54:20.511174Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 8.0.26, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2022-08-29T18:54:20.511404Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2022-08-29T18:54:20.704305Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-08-29T18:54:20.704641Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2022-08-29T18:54:20.704677Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-08-29T18:54:20.705483Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30)  MySQL Community Server - GPL.

I think the issue is related to using the same pv which left few residues from the previous installation in /var/lib/sql. Can you try the solution in this comment here and let me know if it works for you? I'm not confident though as i see innodb_fast_shutdown set to 2.

Just in case you are wondering how to do it - You can attach a pod to katib-mysql pvc and perform the operation or temporarily update the sql container command to sleep and do a kubectl exec on it.

[pablofiumara]

@midhun1998 Thanks again. I tried and found no /var/lib/mysql/ib_logfile* files

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

Feel free to re-open an issue if you have any followup problems.