vitess-operator icon indicating copy to clipboard operation
vitess-operator copied to clipboard

Bug Report: Keyspace-Level Image Settings Issue in Vitess Operator

Open yydoow opened this issue 6 months ago • 3 comments

Bug Report: Keyspace-Level Image Settings Issue in Vitess Operator

Description

We’ve identified a bug in Vitess Operator v18 related to keyspace-level image settings. The issue occurs when keyspaces are sorted alphabetically. If one keyspace is configured to use the mysql80Compatible image, all keyspaces that come after it (in alphabetical order) will incorrectly use the MySQL 8.0 image, even if they are supposed to default to mysql56Compatible. Keyspaces that appear earlier in the alphabetical order are unaffected and continue using the mysql56Compatible image as expected.

Expected Behavior

Keyspaces without keyspace-level image settings should default to the mysql56Compatible image (18.0.7-percona57-hc-cfc6d4900a), regardless of their alphabetical position relative to keyspaces configured with mysql80Compatible.

Observed Behavior

Keyspaces without keyspace-level image settings are incorrectly using the mysql80Compatible image (18.0.7-percona80-hc-635e8e91b7) for the mysqld container.

Steps to Reproduce

Create multiple keyspaces in Vitess Operator v18, ensuring they are sorted alphabetically. Configure one keyspace with the mysql80Compatible image setting. Leave other keyspaces without keyspace-level image settings (defaulting to mysql56Compatible). Observe that keyspaces following the mysql80Compatible keyspace in alphabetical order incorrectly use the MySQL 8.0 image for their mysqld containers.

Examples

Keyspace Here is an example of a VitessKeyspace CR that does not have keyspace-level image settings but is incorrectly using the mysql80Compatible image:

apiVersion: planetscale.com/v2
kind: VitessKeyspace
metadata:
  name: xxxxx
  creationTimestamp: "2025-06-13T06:38:25Z"
images:
  mysqld:
    mysql80Compatible: docker.ihs.xxx.net/platform-dbtech/vitess-lite:18.0.7-percona80-hc-635e8e91b7
  mysqldExporter: docker-hub.ihs.xxx.net/prom/mysqld-exporter:v0.11.0
  vtbackup: docker.ihs.xxx.net/platform-dbtech/vitess-lite:18.0.7-percona57-hc-cfc6d4900a
  vtorc: docker.ihs.xxx.net/platform-dbtech/vitess-lite:18.0.7-percona57-hc-cfc6d4900a
  vttablet: docker.ihs.xxx.net/platform-dbtech/vitess-lite:18.0.7-percona57-hc-cfc6d4900a

Pod The following pod description highlights the issue. The mysqld container is incorrectly using the mysql80Compatible image (18.0.7-percona80-hc-635e8e91b7) instead of the expected mysql56Compatible image (18.0.7-percona57-hc-cfc6d4900a):

Containers:
  mysqld:
    Image: docker.ihs.xxx.net/platform-dbtech/vitess-lite:18.0.7-percona80-hc-635e8e91b7
    Environment:
      MYSQL_FLAVOR: MySQL80

yydoow avatar Jun 30 '25 01:06 yydoow

This bug exists in main and we are able to repro this easily. Define 4 new keyspaces in 101_initial_cluster.yaml, example_a, example_b, example_c & example_d Override the images: for example_b, set the vttablet: & mysqld.mysql80Compatible: to something different. Apply and inspect the vitesskeyspaces objects, You will notice that the example_b reflects the different vttablet & mysqld images. But example_c & example_d also shows the mysqld image change (leaking via the example_b setting). their vttablet image remains intact.

Summary: mysqld: image tag override is being wrongfully applied to rest of the keyspaces in the list.

bluecrabs007 avatar Jul 03 '25 23:07 bluecrabs007

Issue Reproduction

This error is very easy to reproduce.
Here are the results:

Result Screenshot

I also conducted an experiment and found that it only affects mysql80Compatible, but not vttablet.

Experiment Screenshot

Git Diff

Here is the git diff showing the changes made:

GrumpyCat0x avatar Jul 03 '25 23:07 GrumpyCat0x

@yydoow @bluecrabs007, thank you for reporting this issue. I have opened a PR to fix it: https://github.com/planetscale/vitess-operator/pull/713.

frouioui avatar Jul 30 '25 21:07 frouioui