ddev fix: Use bitnami/mysql as base for mysql:8.x images, and support MySQL 8.4, fixes #6241

The Issue

I don't expect this to be completed right away, but mysql:8.0 and mysql:8.4 are now possible with this. The builds for all other images are broken by it though :)

Fixes

#6241
https://github.com/ddev/ddev/issues/6511

Replaces

https://github.com/rfay/ddev/pull/49 - moving this to ddev/ddev for visibility as it should be a good effort in coming weeks or months.

How This PR Solves The Issue

There are a number of outstanding issues here, but I've made some progress:

[ ] mysql:5.7 will have to continue to be built from ddev/mysql:5.7, which will still have to be maintained by us: xtrabackup 2.4 will never be available on arm64 from Percona, so our mysql:5.7 support will apparently have to continue to be built on ubuntu:18.04, where both mysql-server (5.7) and percona-xtrabackup are available. There once was a percona-xtrabackup in Debian snapshot but it was v2.2.3 and based on mysql 5.6. The one that we get with ddev/mysql:5.7 (from ubuntu:18.04) is "2.4.9 based on MySQL server 5.7.13".
[x] mysql:8.0 is based on bitnami: I think that when Percona Xtrabackup rolls out their 8.0 arm64 support that we'll be able to build ddev-dbserver for mysql 8.0 based on bitnami/mysql:8.0, which will be a big win getting rid of our xtrabackup-build and probably free us from having to pin matching xtrabackup/mysql-server versions.
[x] I haven't tried building mariadb versions from bitnami/mariadb, and won't try it in this PR.

Outstanding significant problems (besides waiting for arm64 xtrabackup)

[ ] The bitnami config files in /opt/bitnami override what we put in /etc/my.cnf. AFAICT this is a result of how mysqld is built. This has to be investigated, and we either have to override what they provide or use their docker-entrypoint.sh or something.
[ ] We need separate my.cnf (or other config file) for different versions. We've used a common config for all maria/mysql for a long time, but now mysql:8.0 whines about deprecations and mysql:8.4 actually implements them.
[ ] The bitnami image has done something to make $HOME not be set. As a hack, I set it in app_compose_template.yaml, but probably need to figure out why it doesn't get set. We could also set it in our code that adds the user layer to the image.
[ ] Review TODOs here and make sure they're removed/accounted for
[ ] Our developer docs about the build process will need to be updated, removing the xtrabackup-build, but keeping the ddev/mysql:5.7, etc. xtrabackup-build can hopefully be archived.

Manual Testing Instructions

[ ] Use mysql:8.4 and mysql:8.0 in both arm64 and amd64 environments.

Automated Testing Overview

Release/Deployment Notes

Oct 06 '24 16:10 rfay

Download the artifacts for this pull request:

See Testing a PR.

Oct 06 '24 16:10 github-actions[bot]

At this point we can build the mysql:8.0 and 8.4 images, but everything else fails:

https://github.com/rfay/ddev/actions/runs/11203465444

Oct 06 '24 17:10 rfay

If you wouldn't mind taking a first look at this @stasadev I'd appreciate it. I'm hoping the main risks are mitigated, and I think we get rid of a lot of complex maintenance. But I haven't compared import performance, and turning off binlogs is I think a good thing but it is a bit of a change.

Dec 12 '24 04:12 rfay

using dump from DDEV v1.24.1 and mysql:8.0:

✅ ddev delete -Oy && ddev config --database=mysql:8.0 && ddev import-db --file=/path/to/dump.sql.gz
✅ ddev debug migrate-database mysql:8.4
✅ ddev delete -Oy && ddev config --database=mysql:8.4 && ddev import-db --file=/path/to/dump.sql.gz
✅ ddev snapshot and ddev snapshot restore - tested both 8.0 and 8.4
✅ large dump ddev delete -Oy && ddev config --database=mysql:8.4 && ddev import-db --file=/path/to/dump.sql.gz

Dec 16 '24 17:12 stasadev

I compared ddev import-db for mysql:8.0 dump (22 GB unzipped) and tested it on --database=mysql:8.0 in v1.24.1 and in this PR, and got the same import time ~75 minutes.

However, when I tried to do the same ddev import-db using --database=mysql:8.4, it was completed in ~95 minutes. Not sure if it's relevant (because the dump was done with mysql:8.0 before).

It's probably related to this https://perconadev.atlassian.net/browse/PS-9228 and not related to DDEV.

Dec 18 '24 15:12 stasadev

Wow, more than a year after starting on it. Followup will be using the production repo when they get it working, and there are perhaps docs, but we don't have to do the php7.3 docs any more I guess.

Dec 21 '24 17:12 rfay