Parallelize Shard Rebalancing & Unlock Concurrent Logical Shard Moves

[codeforall]

This pull request delivers a major speedup for cluster maintenance by parallelizing shard rebalancing and eliminating the bottlenecks that prevented concurrent logical-replication moves. Together, these changes shorten rebalance windows—especially for clusters with large reference tables—and unblock multiple shard transfers from running side-by-side.

[emelsimsek]

I suggest some refactoring. This way logical replication mode can be handled as well.

Current code has one function TransferShards, that copies all the colocated shards sequentially and then creates the constraints and updates the metadata.

Now since we are moving to parallel copy of those shards, we can refactor TransferShards into two functions:

CopyShardData(shardid)

         Insert cleanup records
	 Create shard table on worker
	 CopyShardsToNode 

         If logical replication
               catch-up
               Lock Shard
         
          <-- Shard is locked for blocking_writes mode --->
 
CompleteShardTransferForColocation(shardid)

        Create constraints, indexes, FKs
        Update pg_dist_placement
        Update metadata in other workers.

        Unlock shards

The corresponding UDFs can be then scheduled as tasks in the background.

For instance, for moving ref1, ref2 and ref3, background task queue would look like this.

task1 = Copy_shard(ref1) task2 = Copy_shard(ref2) task3 = Copy_shard(ref3)

CompleteShardTransferForColocation(ref1) depends on {task1,task2,task3}

[codecov[bot]]

Codecov Report

:x: Patch coverage is 0.37879% with 263 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 45.26%. Comparing base (2095679) to head (8615ac2). :warning: Report is 1 commits behind head on main.

:x: Your patch status has failed because the patch coverage (0.37%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage. :x: Your project status has failed because the head coverage (45.26%) is below the target coverage (87.50%). You can increase the head coverage or adjust the target coverage.

:exclamation: There is a different number of reports uploaded between BASE (2095679) and HEAD (8615ac2). Click for more details.

HEAD has 53 uploads less than BASE

Flag	BASE (2095679)	HEAD (8615ac2)
15_regress_check-pytest	1	0
17_regress_check-pytest	1	0
16_regress_check-pytest	1	0
15_citus_upgrade	1	0
16_regress_check-follower-cluster	1	0
17_regress_check-follower-cluster	1	0
17_regress_check-columnar-isolation	1	0
16_regress_check-columnar-isolation	1	0
15_regress_check-columnar-isolation	1	0
15_regress_check-follower-cluster	1	0
15_regress_check-enterprise-isolation-logicalrep-2	1	0
16_regress_check-query-generator	1	0
16_regress_check-enterprise-failure	1	0
17_regress_check-query-generator	1	0
17_regress_check-enterprise-failure	1	0
16_regress_check-split	1	0
15_regress_check-enterprise-failure	1	0
17_regress_check-split	1	0
15_regress_check-columnar	1	0
15_regress_check-enterprise-isolation	1	0
15_regress_check-split	1	0
16_regress_check-enterprise-isolation-logicalrep-1	1	0
17_regress_check-enterprise-isolation-logicalrep-1	1	0
16_regress_check-enterprise-isolation	1	0
17_regress_check-enterprise-isolation	1	0
15_regress_check-enterprise-isolation-logicalrep-1	1	0
16_regress_check-failure	1	0
16_arbitrary_configs_3	1	0
15_regress_check-multi-mx	1	0
15_arbitrary_configs_1	1	0
15_arbitrary_configs_4	1	0
17_arbitrary_configs_2	1	0
15_regress_check-multi-1	1	0
16_regress_check-multi	1	0
15_arbitrary_configs_0	1	0
15_regress_check-multi	1	0
17_regress_check-multi	1	0
17_regress_check-multi-1	1	0
16_regress_check-multi-1	1	0
17_arbitrary_configs_4	1	0
16_arbitrary_configs_0	1	0
17_arbitrary_configs_0	1	0
16_arbitrary_configs_1	1	0
16_regress_check-columnar	1	0
17_regress_check-columnar	1	0
16_regress_check-vanilla	1	0
15_regress_check-vanilla	1	0
15_regress_check-query-generator	1	0
17_regress_check-enterprise	1	0
15_regress_check-enterprise	1	0
17_regress_check-vanilla	1	0
16_regress_check-enterprise	1	0
17_arbitrary_configs_3	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #7983       +/-   ##
===========================================
- Coverage   87.59%   45.26%   -42.34%     
===========================================
  Files         285      285               
  Lines       62252    61559      -693     
  Branches     7818     7630      -188     
===========================================
- Hits        54529    27863    -26666     
- Misses       5386    31157    +25771     
- Partials     2337     2539      +202     

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tejeswarm]

@codeforall No major code-comments. I need to understand some nuances; can we have a short call on your Monday morning to finalize?