docs icon indicating copy to clipboard operation
docs copied to clipboard
verified publisher icon pingcap

DM: Consider not putting `--consistency none` in its sample task config

Open kennytm opened this issue 6 months ago • 4 comments

Change Request

Please answer the following questions before submitting your issue. Thanks!

  1. Describe what you find is inappropriate or missing in the existing docs.

https://docs.pingcap.com/tidb/stable/task-configuration-file-full/#task-configuration-file-template-advanced

  1. Describe your suggestion or addition.

The sample config contained this fragment:

mydumpers:
  global:                            # The configuration name of the processing unit.
    threads: 4                       # The number of threads that access the upstream when the dump processing unit performs the precheck and exports data from the upstream database (4 by default)
    chunk-filesize: 64               # The size of the file generated by the dump processing unit (64 MB by default).
    extra-args: "--consistency none" # Other arguments of the dump processing unit. You do not need to manually configure table-list in `extra-args`, because it is automatically generated by DM.

The problem is that --consistency none will generate an inconsistent dump of the source data, this caused 2 problems:

  1. The dump may contain rows with duplicated PKs, which may forces the "load" step to employ conflict resolution which slows down the procedure.
  2. The subsequent "sync" step also needs to stay in safe-mode for much longer time.

This --consistency none is only needed for upstream which does not provide table-locking command/privileges such as Aurora. So users who just copy the sample without thinking much will have suboptimal performance at the start.

This extra-args content should better be changed to something that has less impact.

  1. Provide some reference materials (such as documents and websites) if you could.

(internal reference TICKET-6178)

kennytm avatar Jun 13 '25 03:06 kennytm

@River2000i Could you please help take a look at this and submit a PR if changes are needed?

Oreoxmt avatar Jun 13 '25 04:06 Oreoxmt

For the template config file of DM, i think --consistency none just an example to show how to set arguments for dump unit. I will update it.

JQWong7 avatar Jun 13 '25 05:06 JQWong7

@kennytm How about we keep the --consistency, but using default value --consistency auto

JQWong7 avatar Jun 18 '25 03:06 JQWong7

@River2000i LGTM

kennytm avatar Jun 18 '25 08:06 kennytm