[DocDB] Fix scans that do not honor timeouts in certain scenarios, to not run into runaway scan scenarios.

[rthallamko3]

Jira Link: DB-10731

Description

#569 added the capability to honor timeout by DocDB Range Scan Queries, however if the scan involves scanning a large number of intents, then the timeout is not honored leading to run away scans that end up eating a core.

If there are large number of scans running (as many as the number of cores on the node), it can run into 100% CPU usage too.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.

[hari90]

here is simple repro: https://phorge.dev.yugabyte.com/D33833

[yusong-yan]

Root cause of the read timeout issue(existed since day 0)

Current read deadline is only checked when processing subdocs of a row. However, in certain cases, before we even process subdocs, the code might skip the row (code link), meaning read deadline will not be checked. If this frequently occurs during a read operation, the read can continue indefinitely even after the read deadline has passed.

Queries That Can Lead to This Issue

This issue can occur when reading through a non-prefix of an index key. For an index with a key structure of (k1, k2, k3, k4, …), a query that sets a non-prefix of these keys could cause this problem: SELECT * FROM table WHERE k1 = x AND k3 = y SELECT * FROM table WHERE k1 = x AND k4 = y

Here is a example:

CREATE TABLE test_table (custid text, acctid int, roletitle text, PRIMARY KEY (custid, acctid, roletitle) ) WITH transactions = { 'enabled' : true };

INSERT INTO test_table (custid, acctid, roletitle) VALUES ('cust123', 1, 'Manager');
INSERT INTO test_table (custid, acctid, roletitle) VALUES ('cust123', 2, 'Developer');
INSERT INTO test_table (custid, acctid, roletitle) VALUES ('cust123', 3, 'Developer');
INSERT INTO test_table (custid, acctid, roletitle) VALUES ('cust123', 4, 'Developer');

select acctid from test_table where custid = 'cust123' AND roletitle = 'Manager';

After reading the Manager row, we then walk through the next three rows without checking deadline.