[Dynamic Regions] raftstore v2: TiKV has inconsistent index

[CalvinNeo]

Bug Report

Ignore index

mysql> select /*+ read_from_storage(tikv[db12.other_handle7]) ignore_index(db12.other_handle7,PRIMARY) */ count(*) from db12.other_handle7 as of timestamp '2023/08/28 04:15:21';
+----------+
| count(*) |
+----------+
| 12100000 |
+----------+
1 row in set (0.57 sec)

Using index, the result fluctuates, not equal to table scan

mysql> select /*+ read_from_storage(tikv[db12.other_handle7]) use_index(db12.other_handle7,PRIMARY) */ count(*) from db12.other_handle7 as of timestamp '2023/08/28 04:15:21';
+----------+
| count(*) |
+----------+
| 12099999 |
+----------+
1 row in set (0.78 sec)
mysql> select /*+ read_from_storage(tikv[db12.other_handle7]) use_index(db12.other_handle7,PRIMARY) */ count(*) from db12.other_handle7 as of timestamp '2023/08/28 04:15:21';
+----------+
| count(*) |
+----------+
| 11870175 |
+----------+
1 row in set (0.83 sec)
mysql> select /*+ read_from_storage(tikv[db12.other_handle7]) use_index(db12.other_handle7,PRIMARY) */ count(*) from db12.other_handle7 as of timestamp '2023/08/28 04:15:21';
+----------+
| count(*) |
+----------+
| 11805905 |
+----------+
1 row in set (0.76 sec)

What version of TiKV are you using?

What operating system and CPU are you using?

Steps to reproduce

What did you expect?

consistent index and table scan

What did happened?

inconsistent index and table scan

[overvenus]

AS OF TIMESTAMP clause enables stale read and you may retrieve "stale" data, leading to potential inconsistencies in the results.

See https://docs.pingcap.com/tidb/stable/stale-read

[CalvinNeo]

AS OF TIMESTAMP clause enables stale read and you may retrieve "stale" data, leading to potential inconsistencies in the results.

See https://docs.pingcap.com/tidb/stable/stale-read

The document says

Specifying an exact point in time (recommended): If you need TiDB to read data that is globally consistent from a specific point in time without violating the isolation level, you can specify the corresponding timestamp of that point in time in the query statement.

So, IMO stale read by AS OF TIMESTAMP clause won't do harm to consistency issue.

Meanwhile, the system is not even eventual consistent, since the result from index is not consistent after a long time the wordload ends.

[tonyxuqqi]

What are the steps for repro @CalvinNeo Is TiFlash involved?

[CalvinNeo]

What are the steps for repro @CalvinNeo Is TiFlash involved?

This problem can be reproduced by a certain consistency test of TiFlash. We force the consistency test to read from tikv without index in order to make it run further.

I think it is some problem relate to index.

[zhangjinpeng87]

Close this issue temporarily since Dynamic Regions project is paused.