Partial Movetables: allow moving a table one shard at a time

[rohit-nayak-ps]

Warning!  This feature is expected to be used only for migrating large Vitess clusters across data centers and should not be enabled on an ongoing basis since it adds performance overhead during query serving. It also changes the paradigm hitherto expected in Vitess that a table is served, at any given time, from a single keyspace. The latter might lead to unstable behavior, with global routing, for example.

TL;DR;

• MoveTables can now take a subset of shards
• ShardRoutingRules have been added, to route shards already moved from source to target
• ShardRoutingRules apply both for shard targeted and globally routed queries

Description

This feature introduces the concept of partial keyspaces where some shards are served from a different keyspace. This is useful for a specific but critical use-case where a large production Vitess setup (100s of shards) is being migrated to a new data center or provider. Migrating the entire cluster in one go using MoveTables could cause an unacceptable downtime due to the large number of primaries that need to be synced when writes are switched.

Sample Usage

partial MoveTables signalled by --source_shards

vtctlclient MoveTables -- -source customer --tables 'customer,corder' --source_shards '-80' Create customer2.partial1

VDiff works as-is

vtctlclient VDiff customer2.partial1

SwitchTraffic generates this shard routing rule

vtctlclient MoveTables -- SwitchTraffic customer2.partial1

{"rules":[{"from_keyspace":"customer", "to_keyspace":"customer2", "shards":"-80"}]}

Demo that shard routing now works

# stop workflow to stop reverse replication from running 
vtctlclient Workflow customer2.partial stop
# update customer2 database directly.
mysql -S $VTDATAROOT/vt_0000000500/mysql.sock -u vt_dba -e "update vt_customer2.customer set email = concat('routed.', email)"

# use shard targeting using customer (not customer2). 
mysql -e "use customer:-80; select customer_id, email from customer order by customer_id"

+-------------+---------------------------+
| customer_id | email                     |
+-------------+---------------------------+
|           1 | [email protected]   |
|           2 | [email protected]     |
|           3 | [email protected] |
|           5 | [email protected]     |
+-------------+---------------------------+

Summary of code changes

Core Changes

Workflow Show changes

While we had the possibility of partial reads being switched earlier, now writes can also be partially switched in a workflow.

Shard Routing Rules

Topo

Shard Routing Rules are a new concept introduced for this feature. It maps a (keyspace, shard) tuple to another keyspace. This is used to create a new cluster level map which maps a (keyspace, shard) to another keyspace. These are set by SwitchTraffic in a partial MoveTables and used by vtgate while routing queries. vtctlclient commands ApplyShardRoutingRules and GetShardRoutingRulesallow setting/getting of these rules.

vtgate Shard Targeting

The shard targeted query routing in vtgate's bypass mechanism go/vt/vtgate/planbuilder/bypass.go. We create a map from the SrvVSchema's shard routing rules object and check if a specified shard destination needs to be rerouted.

vtgate Global Routing

The global query routing using vtgate's ResolveDestinations() methods go/vt/vtgate/vcursor_impl.go. We go through all selected shard destinations and modify those that are mapped in the shard routing rules.

New column workflow_sub_type in _vt.vreplication

There is a new bool column workflow_sub_type added to _vt.vreplication, set for partial movetables. It is used for visibility and for bypassing certain validations that expect a full keyspace.

Flags

vtgate --enable_shard_routing

Default: false. It is used when a cluster is setup with shard routing rules and tells vtgate to use these rules while routing queries.

MoveTables --source_shards

The only flag needed to tell MoveTables to perform a partial movetables is to pass it this flag, example --source_shards -80. This flag already exists and is used by Reshard.

Notes

Both read and write traffic is switched at the same time when a shard routing is deemed complete (using SwitchTraffic). This is because we add a shard routing rule when this happens. Switching of read and write separately is done by updating the regular routing rules by targeting @replica and @rdonly.

Test Changes

e2e test TestPartialMoveTables

It creates a workflow which moves tables from one shard only to the target shard, ensures it completed correctly and switches traffic. It ensures that the shard routing rules and regular routing rules are setup correctly and that the vtgate queries both shard targeted and global queries are routed correctly.

Modified vtgate tests

A new set of tests have been added where the test cluster is setup as a partial cluster: one shard is in a different keyspace with the ShardRoutingRules setup. The main change is that the vtgate params need to specify the default keyspace as the DbName to avoid ambiguity.

Some tests have to be skipped for partial keyspaces because they intrinsically affect tablets in the base cluster which are now not all serving.

TODOs: [ ] Add explicit reasons for Skipped vtgate unit tests during partial keyspace

Checklist

• [ ] Should this PR be backported?
• [X] Tests were added or are not required
• [ ] Documentation was added or is not required

[rohit-nayak-ps]

Squashed all commits since it was becoming tougher to fix conflicts each time with a large number of commits..

[vitess-bot[bot]]

Review Checklist

Hello reviewers! :wave: Please follow this checklist when reviewing this Pull Request.

General

• [ ] Ensure that the Pull Request has a descriptive title.
• [ ] If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
• [ ] If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive. Additionally, flag names should use dashes (-) as word separators rather than underscores (_).
• [ ] If a workflow is added or modified, each items in Jobs should be named in order to mark it as required. If the workflow should be required, the GitHub Admin should be notified.

Bug fixes

• [ ] There should be at least one unit or end-to-end test.
• [ ] The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

• [ ] There should be some code comments as to why things are implemented the way they are.

New/Existing features

• [ ] Should be documented, either by modifying the existing documentation or creating new documentation.
• [ ] New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• [ ] Protobuf changes should be wire-compatible.
• [ ] Changes to _vt tables and RPCs need to be backward compatible.
• [ ] vtctl command output order should be stable and awk-able.