br
br copied to clipboard
Lightning imports data to non-destination cluster and leads to index inconsistency with data
Please answer these questions before submitting your issue. Thanks!
- What did you do? If possible, provide a recipe for reproducing the error.
mini reproduce:
deploy two tidb cluster using different status-port:
import data using tidb-lightning local mode
sysbench /usr/share/sysbench/oltp_common.lua --mysql-host=127.0.0.1 --mysql-port=6000 --mysql-db=test --mysql-user=root --mysql-password= --table_size=2000 --tables=1 --events=10000 --report-interval=10 --time=0 prepare
sysbench /usr/share/sysbench/oltp_common.lua --mysql-host=127.0.0.1 --mysql-port=4000 --mysql-db=test --mysql-user=root --mysql-password= --table_size=200 --tables=1 --events=10000 --report-interval=10 --time=0 prepare
tiup dumpling -B test -o ./sbtest1_dump -P 6000
mysql -u root -h 127.0.0.1 -P 6000 test
truncate table sbtest1;
[tidb@172-16-120-122 ~]$ tiup tidb-lightning -c lightning.toml
tiup is checking updates for component tidb-lightning ...
Starting component `tidb-lightning`: /home/tidb/.tiup/components/tidb-lightning/v6.1.0/tidb-lightning -c lightning.toml
Verbose debug logs will be written to tidb-lightning.log
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| # | CHECK ITEM | TYPE | PASSED |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 1 | Source csv files size is proper | performance | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 2 | checkpoints are valid | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 3 | table schemas are valid | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 4 | Cluster is available | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 5 | Lightning has the correct storage permission | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 6 | sorted-kv-dir:./some_dir and data-source-dir:/home/tidb/sbtest1_dump are in the same disk, may slow down performance | performance | false |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 7 | local disk resources are rich, estimate sorted data size 383.1KiB, local available is 806.2GiB | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 8 | Cluster available is rich, available is 10.77TiB, we need 1.122MiB | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 9 | Cluster doesn't have too many empty regions | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 10 | Cluster region distribution is balanced | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
Error: [Lighting:Restore:ErrChecksumMismatch]checksum mismatched remote vs local => (checksum: 2717476634684440220 vs 6245448145285911419) (total_kvs: 4200 vs 4000) (total_bytes:511600 vs 504000)
tidb lightning encountered error: [Lighting:Restore:ErrChecksumMismatch]checksum mismatched remote vs local => (checksum: 2717476634684440220 vs 6245448145285911419) (total_kvs: 4200 vs 4000) (total_bytes:511600 vs 504000)
mysql -u root -h 127.0.0.1 -P 4000 test
mysql> admin check table sbtest1;
ERROR 8134 (HY000): data inconsistency in table: sbtest1, index: k_1, col: k, handle: "1", index-values:"KindInt64 100" != record-values:"KindInt64 999", compare err:<nil>
a. tiup tidb-lightning -c lightning.toml
Error: [Lighting:Restore:ErrChecksumMismatch]checksum mismatched remote vs local => (checksum: 2717476634684440220 vs 6245448145285911419) (total_kvs: 4200 vs 4000) (total_bytes:511600 vs 504000)
b. mysql> admin check table sbtest1;
ERROR 8134 (HY000): data inconsistency in table: sbtest1, index: k_1, col: k, handle: "1", index-values:"KindInt64 100" != record-values:"KindInt64 999", compare err:<nil>
- What did you expect to see?
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
| # | CHECK ITEM | TYPE | PASSED |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
......
| 4 | Cluster is available and correct | critical | true |
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
......
+----+----------------------------------------------------------------------------------------------------------------------+-------------+--------+
- What did you see instead?
1. | 4 | Cluster is available | critical | true |
2. checksum mismatched remote vs local
3. data inconsistency in table (in the non-destination cluster)
- What version of BR and TiDB/TiKV/PD are you using?
tidb-lightning v6.1.0 tidb-server v6.1.0、v6.2.0-alpha tikv-server v6.1.0 pd-server v6.1.0
-
Operation logs
- Please upload
br.log
for BR if possible - Please upload
tidb-lightning.log
for TiDB-Lightning if possible - Please upload
tikv-importer.log
from TiKV-Importer if possible - Other interesting logs
- Please upload
-
Configuration of the cluster and the task
-
tidb-lightning.toml
for TiDB-Lightning if possible
-
[tidb@172-16-120-122 ~]$ cat lightning.toml
[lightning]
level = "info"
file = "tidb-lightning.log"
check-requirements = true
[mydumper]
data-source-dir = "./sbtest1_dump"
[tikv-importer]
backend = "local"
sorted-kv-dir = "./some_dir"
[tidb]
host = "127.0.0.1"
port = 6000
user = "root"
password = ""
status-port = 10080
- Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible
/assign