Slow message handling: `Tick(RaftLogGc)` takes 2.8ms

[overvenus]

Development Task

[2025/06/11 02:24:11.712 +00:00] [WARN] [peer.rs:750] ["dbg slow peer message handling"] [elapsed=2.869589ms] [msg=Tick(RaftLogGc)] [peer_id=273809] [region_id=273807] [thread_id=105]

The log is taken from a TiKV instance that runs 238k peers and manages 231k sst files.

Cluster Details

The TiKV is built from https://github.com/overvenus/tikv/tree/log-slow-message

TiKV configuration:

      [coprocessor]
        region-split-size = "2MiB"

      [log]
        [log.file]
          filename = "/var/lib/tikv/log/tikv.log"
          max-backups = 100
          max-size = 300

      [rocksdb]
        max-manifest-file-size = "256MiB"
        [rocksdb.defaultcf]
          compaction-guard-max-output-file-size = "2MiB"
          compaction-guard-min-output-file-size = "1MiB"
          target-file-size-base = "1MiB"
        [rocksdb.writecf]
          compaction-guard-max-output-file-size = "2MiB"
          compaction-guard-min-output-file-size = "1MiB"
          target-file-size-base = "1MiB"

      [server]
        snap-min-ingest-size = "2KiB"

      [storage]
        reserve-space = "0MB"

Test scripts:

# Prepare
sysbench --db-driver=mysql --mysql-host=tc-tidb-peer --mysql-port=4000 --mysql-user=root         --mysql-db=sbtest --tables=35 --table-size=40351404 --time=3600 --report-interval=3 --threads=20 --auto_inc=true         --create_secondary=off --non_index_updates=10 --mysql-ignore-errors=1105 oltp_update_non_index prepare

# Run
sysbench --db-driver=mysql --mysql-host=tc-tidb-peer --mysql-port=4000 --mysql-user=root         --mysql-db=sbtest --tables=35 --table-size=40351404 --time=3600 --report-interval=3 --threads=200 --auto_inc=true         --create_secondary=off --non_index_updates=10 --mysql-ignore-errors=1105 oltp_update_non_index run