mysql issue on database-0 pod

[ryzior]

Hi I'm having issues with database-0 pod when bootstrapping the whole environment. I was unable to find any relevant information yet and maybe I'm missing some initial env setup? I'm runing k8s 1.20.1 on ubuntu.

 kubectl get pods -n kubecf
NAME                                     READY   STATUS             RESTARTS   AGE
cf-apps-dns-59f9f659f5-kg2jd             1/1     Running            0          8m51s
coredns-quarks-6db68476bd-dvxw6          1/1     Running            0          7m28s
coredns-quarks-6db68476bd-wb7p7          1/1     Running            0          7m28s
database-0                               0/2     CrashLoopBackOff   12         7m8s
database-seeder-a8e75e61ba456096-twfzf   1/2     NotReady           0          7m37s
ig-e5540e5dcfc6e9aa-7xtlg                0/19    Pending            0          3m4s
ig-e5540e5dcfc6e9aa-lk9pg                0/19    Evicted            0          7m17s

container logs:

k logs database-0 database -n kubecf
Removing pending files in /var/lib/mysql, because sentinel was not reached
Running --initialize-insecure on /var/lib/mysql
total 8.0K
drwxrwxrwx 2 65534 65534 4.0K Jan 11 08:22 .
drwxr-xr-x 1 root  root  4.0K Aug 19 17:45 ..
2021-01-11T08:23:03.543324Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-01-11T08:23:03.546531Z 0 [Warning] WSREP: Node is running in bootstrap/initialize mode. Disabling pxc_strict_mode checks
2021-01-11T08:23:03.566633Z 0 [ERROR] **Can't change data directory owner to mysql**
2021-01-11T08:23:03.566664Z 0 [ERROR] Aborting

PV is a NFS share (RW permissions confirmed) - this is just a lab setup.

Here I found this so called bug for mysql: https://bugs.mysql.com/bug.php?id=78591 but the conclussion is that the issue is more related to apparmour config since mysql has been fixed for this issue:

[4 May 2016 8:40] Lars Tangvald The cause was that mysqld would attempt to set ownership on an empty datadir whether or not it was already correct, and the server's apparmor profile will not allow this operation. So this will still fail if the permissions are incorrect on systems with apparmor running, but this is by design.

I did a test on minikube windows and no issue like above for this pod:

c:\bin>kubectl logs database-0 database -n kubecf Removing pending files in /var/lib/mysql, because sentinel was not reached Running --initialize-insecure on /var/lib/mysql total 8.0K drwxrwxrwx 2 root root 4.0K Jan 13 12:57 . drwxr-xr-x 1 root root 4.0K Aug 19 17:45 .. Finished --initialize-insecure MySQL init process in progress... There were fatal errors during processing of zoneinfo directory

MySQL init process done. Ready for start up.

Maybe there are some storage requirements I have to fulfill, I'm using NFS share and 2 PVs created based on PVC existing in the system.