[DocDB] Master is attempting to a delete a tablet that is being bootstrapped

[lingamsandeep]

Jira Link: DB-10700

Description

• During scale down and perhaps even scale up, we see that some tablets are moved from ABC -> ABD
• D reports this tablet as part of its TabletReport - indicating a consensus::kInvalidOpIdIndex
• When this happens, the eviction of tombstone replicas logic kicks in and sends a DeleteReplica to the T-server 'D' here: https://github.com/yugabyte/yugabyte-db/blob/030ae0705569e5afb997e49a6dd959115aefaaf1/src/yb/master/catalog_manager.cc#L7485
• The Delete itself fails on the T-server as the Tablet is in transition, but this causes a bunch of back and forth message exchanges.
• As long as there is a DELETE pending, we don't do any more ADDs to it (https://github.com/yugabyte/yugabyte-db/issues/21806)
• FLAGS_master_tombstone_evicted_tablet_replicas controls this behavior of eviction.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.