MDEV-34431: Avoid spin loops on page I/O waits

[dr-m]

• [x] The Jira issue number for this PR is: MDEV-34431

Description

While waiting for for I/O completion, let us skip spin loops. Even on fast storage, reading a page into the buffer pool takes so long time that wasting CPU in a spin loop does not make any sense.

block_lock::s_lock_nospin(): A no-spin variant of acquiring a shared buffer page latch. Regular s_lock() always involveis a spin loop.

ssux_lock_impl::rd_lock_spin(), ssux_lock_impl::rd_lock_nospin(): Split from rd_wait().

ssux_lock_impl::rd_lock(): Invoke either rd_lock_nospin() or rd_lock_try() and rd_lock_spin().

The above changes are also part of #4168.

buf_read_page(): Return a pointer to a buffer-fixed, non-read-fixed page, or nullptr in case of an error.

buf_inc_get(): Remove the parameter.

IORequest::read_complete(): Assert that the page is both read-fixed and buffer-fixed. Sample recv_sys.recovery_on only once. Buffer-unfix the page when the asynchronous read completes.

buf_page_t::read_complete(): Assert that the page is both read-fixed and buffer-fixed.

buf_page_init_for_read(): Return a pointer to a buffer-fixed block descriptor pointer, bitwise-ORed with 1 in case the block already exists in the buffer pool.

buf_read_ahead_update(), buf_read_ahead_update_sql(): Common code for updating some statistics counters.

buf_read_page_low(): Replace the parameter sync with err, which will return an error code to a synchronous caller. Add a parameter for thread-local mariadb_stats. Return the pointer to the block, or the special values nullptr (read failure) or -1 or -2 for asynchronous reads. Increment the statistics when a synchronous read was requested. In a synchronous read, if the page has already been allocated in the buffer pool but it is read-fixed, wait for the read to complete.

buf_read_page_background(): Update the statistics if supplied.

Release Notes

When a buffer pool lookup needs to wait for page I/O, InnoDB will avoid excessive CPU usage.

How can this PR be tested?

Sysbench or HammerDB on a workload that is larger than the buffer pool. This should be most prominent when there are synchronous waits for pages in the buffer pool.

Measuring the CPU usage while running the test case in MDEV-32067 should show some impact.

Basing the PR against the correct MariaDB version

• [ ] This is a new feature or a refactoring, and the PR is based against the main branch.
• [ ] This is a bug fix, and the PR is based against the earliest maintained branch in which the bug can be reproduced.

The second change (refactoring buf_read_page()) depends on earlier changes f27e9c894779a4c7ebe6446ba9aa408f1771c114 and d6aed21621e438e6135415c0c09fc6adfc0be368, and 11.4 is the earliest maintained branch that contains them.

PR quality check

• [x] I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
• [ ] For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}