mixedversion: introduce preserve_downgrade_option_randomizer mutator

[renatolabs]

This is the first mutator implementation added to the mixedversion framework: it randomizes the point in time when we reset the cluster.preserve_downgrade_option cluster setting in the test.

Fixes: #111595.

Release note: None

[cockroach-teamcity]

This change is 

[renatolabs]

This PR introduces the first official mixedversion plan mutator: preserve_downgrade_option_randomizer. For motivation, see #111595. When this mutator is enabled, the point in which we call RESET CLUSTER SETTING cluster.preserve_downgrade_option is randomized.

I was able to reproduce the issue fixed by #111684 by running acceptance/version-upgrade in local mode enough times for the mutator to be enabled at the "right" time for the bug to manifest. If this was merged today, some upgrade test would likely fail in the PR itself, or shortly after merge given the volume of test runs we get in CI and nightly tests.

Even though there will be some more follow-up work related to mutators, this PR concludes this series of PRs to introduce mutators. Thanks for reviewing!

Context: mixedversion test mutators

This PR is part of a larger effort to introduce the concept of plan mutators to the mixedversion framework. For initial discussion, see internal thread.

In short, mutators are components that modify (i.e., mutate) the original test plan generated by the mixedversion planner. These changes are supposed to introduce new test scenarios in ways that do not interfere with the vast majority of tests. Tests will have the ability to individually enable or disable specific mutators as necessary; by default, all mutators will apply under some probability.

There is a working implementation of the entire idea in the rc/mixedversion-plan-mutators branch, in case it is helpful to see parts of the entire picture. That branch includes the proposed mutator API and an implementation of a mutator that, when merged, will close #111595.

History

• #115118: where we introduced the upgradePlan data structure and allows the upgrade to isolate the steps associated with each upgrade performed during the test.
• #117498: where we removed the ID() function from singleStepProtocol, allowing IDs to be assigned to steps later in the process, after the entire plan has been generated (and, now, mutations have been applied).
• #117993 is one of the pillars of this work, defining the interface to be implemented by mutators, as well as how the mutations they produce are applied to the original test plan.
• #118441 starts paving the way for the first mutators, with an API to disable them (in case a particular mutator is incompatible with a test), and by renaming the finalize step, foreshadowing the first (yet to be introduced) mutator implementation.
• #118666 makes another API change that brings user-provided mixed-version hooks and mixed-version functions implemented by the framework closer together. With those changes, the function to be provided by mutators looks and behaves just like a regular mixed-version hook in a test.

[renatolabs]

TFTR!

bors r=DarrylWong

[craig[bot]]

Build succeeded:

• Bazel Essential CI (Cockroach)

[renatolabs]

No failures on master so far, I think we should be good to backport.

blathers backport 23.2 23.1