root icon indicating copy to clipboard operation
root copied to clipboard

[ntuple] Improve RDF MT scheduling

Open jblomer opened this issue 1 year ago • 1 comments

Implement two improvements to the RNTuple RDF scheduler for the multi-threaded case:

  1. Files of a chain are opened in batches the background: while a batch of files (batch size == number of slots) is processed, the next batch is opened.
  2. For the tail scheduling (multiple slots sharing the same files), the clones of the page source pointing to the same file do not need to re-read meta-data (anchor, header, footer) again.

The patch results in a speed improvement of about 10% for AGC, EOS cached, 32 cores.

jblomer avatar Jun 28 '24 09:06 jblomer

Test Results

    13 files      13 suites   2d 17h 14m 49s :stopwatch:  2 651 tests  2 650 :white_check_mark: 0 :zzz: 1 :x: 32 645 runs  32 644 :white_check_mark: 0 :zzz: 1 :x:

For more details on these failures, see this check.

Results for commit 3d953c9a.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Jun 28 '24 11:06 github-actions[bot]