incubator-pegasus icon indicating copy to clipboard operation
incubator-pegasus copied to clipboard

feat: support to force send non-idempotent write when doing duplication

Open ninsmiracle opened this issue 1 year ago • 2 comments

What problem does this PR solve?

#1907

What is changed and how does it work?

  1. Change the judgment logic in the writing process, and configure it to allow non-idempotent operations in the master cluster.
  2. Modify the logic of the specified mutation committer in prepare_list
  3. Add configuration items and add them to the replica side to modify them through the server-config of admin-cli.
  4. Add the parsing logic after the backup cluster receives the packaged RPC of duplication.
  5. Add the logic for the master cluster to obtain relevant non-idempotent RPC from the plog
  6. Add the metric index of dup non-idempotent write QPS
Tests
  • Unit test
  • Manual test (add detailed scripts or steps below)

when user want to force dup non-idempotent mutations. Type following command in admin-cli:

server-config replica force_send_no_idempotent_when_duplication set true

ninsmiracle avatar Feb 18 '24 12:02 ninsmiracle

This PR's base function is in line with expectations , however I was requested by business department to add more matrix for surveillance system about this feature. I will update this PR in recently days.

ninsmiracle avatar Feb 21 '24 03:02 ninsmiracle

Now , when user config force_send_no_idempotent_when_duplication to true . We can dup non-idempotent write to backup cluster , and we can verify the data consistency by check the number rows of pegasus app. app in master-cluster image

app in backup-cluster image

And because the retries for non-idempotent writes will make influence of the consistency between two clusters. So we should show the metric like following picture: How many non-idempotent write have been retried when doing duplication

show in backup cluster: How many non-idempotent write have been received by backup cluster

ninsmiracle avatar Feb 27 '24 02:02 ninsmiracle