The "CreateWorker" log included the entire DM config unredacted, which leaks secrets into the log

[kennytm]

What did you do?

checked the log, found this kind of log:

[2025/04/02 02:06:06.453 +00:00] [INFO] [master.go:692] [CreateWorker] 
    [job_id=dataflow-engine-job-manager] 
    [worker-type=DMJobMaster] 
    [worker-config="{
        \"seq-id\":204,
        \"created-at\":\"2024-11-12T04:53:28.391Z\",
        \"updated-at\":\"2025-04-02T01:58:23.165Z\",
        \"project-id\":\"1371234567890123456\",
        \"id\":\"345678\",
        \"type\":\"DMJobMaster\",
        \"state\":2,
        \"node-id\":\"dm-job-345678-tiflow-executor-0.tiflow-prod-1371234567890123456-eks-ap-northeast-1-98989898-cfcfcfcf\",
        \"addr\":\"dm-job-345678-tiflow-executor-0.dm-job-345678-tiflow-executor-peer.tiflow-prod-1371234567890123456-eks-ap-northeast-1-98989898.svc:10241\",
        \"epoch\":6679,
        \"config\":\"dGFzay1tb2RlOiBpbmNyZW1lbnRhbApzaGFyZC1tb2RlOiAiIg...\",
        \"error-message\":\"[DFLOW:ErrWorkerSuicide]worker has committed suicide due to master(dataflow-engine-job-manager) having timed out\",
        \"detail\":null,
        \"ext\":{\"selectors\":[{\"label\":\"tiflow-job-to-engine-selector\",\"target\":\"dm-job-345678\",\"op\":\"eq\"}]},
        \"Deleted\":null
    }"] 
    [master-id=dataflow-engine-job-manager]

Decoding the base64-config shows the original DM task config, entirely in plaintext

task-mode: incremental
shard-mode: ""
strict-optimistic-shard-mode: false
ignore-checking-items:
- all
timezone: ""
collation_compatible: loose
target-database:
  host: db-tidb.tidb1379876543210987654.svc
  port: 4000
  user: root
  password: <<TARGET TIDB PASSWORD IN PLAINTEXT>>
  max-allowed-packet: null
  session: {}
  security:
    ssl-ca: ""
    ssl-cert: ""
    ssl-key: ""
    cert-allowed-cn: []
    ssl-ca-bytes: []
    ssl-key-bytes: []
    ssl-cert-bytes: []
    ssl-ca-base64: <<BASE64 OF TLS CA CERT>>
    ssl-key-base64: <<BASE64 OF TLS PRIVATE KEY>> 
    ssl-cert-base64: <<BASE64 OF TLS CERT>>
shadow-table-rules: []
trash-table-rules: []
filters: {}
expression-filter: {}
block-allow-list: {}
mydumpers: {}
loaders: {}
syncers: {}
routes: {}
validators: {}
upstreams:
- source-id: source-987654
  meta:
    binlog-name: mysql-bin-changelog.012345
    binlog-pos: 45678901
    binlog-gtid: ""
  filter-rules: []
  column-mapping-rules: []
  route-rules: []
  expression-filters: []
  black-white-list: ""
  block-allow-list: ""
  mydumper-config-name: ""
  mydumper: null
  mydumper-thread: 0
  loader-config-name: ""
  loader:
    pool-size: 16
    dir: ./dumped_data
    sorting-dir-physical: ""
    import-mode: logical
    on-duplicate: ""
    on-duplicate-logical: replace
    on-duplicate-physical: ""
    disk-quota-physical: 0
    checksum-physical: ""
    analyze: ""
    range-concurrency: 0
    compress-kv-pairs: ""
    pd-addr: ""
  loader-thread: 0
  syncer-config-name: ""
  syncer:
    meta-file: ""
    worker-count: 16
    batch: 100
    queue-size: 1024
    checkpoint-flush-interval: 30
    compact: false
    multiple-rows: false
    max-retry: 0
    auto-fix-gtid: false
    enable-gtid: false
    disable-detect: false
    safe-mode: false
    safe-mode-duration: 60s
    enable-ansi-quotes: false
  syncer-thread: 0
  validator-config-name: ""
  db-config:
    host: upstream-mysql.example.com
    port: 3306
    user: root
    password: <<SOURCE MYSQL PASSWORD IN PLAINTEXT>>
    max-allowed-packet: null
    session: {}
    security: null
  server-id: 0
  flavor: ""
  enable-gtid: false
  case-sensitive: false
experimental:
  async-checkpoint-flush: false
meta-schema: tidbcloud_dm_meta
online-ddl: false
column-mappings: {}
mod-revision: 0

What did you expect to see?

The sensitive fields should be removed from the log:

• target-database.password
• target-database.security.ssl-*
• upstreams[].db-config.password

What did you see instead?

💀

Versions of the cluster

Dataflow Engine version (run tiflow version):

whichever version currently used in tidbcloud

[kennytm]

log location:

https://github.com/pingcap/tiflow/blob/3cf60079e1d73101c376138aed3cb1be2a409704/engine/framework/master.go#L692-L695