root [ntuple] Integrate RNTupleMerger into TFileMerger

This PR integrates recently developed RNTupleMerger (#13858) into TFileMerger so that it can be used via hadd etc. For the time being, the PR is in draft mode to facilitate the discussion.

Jan 19 '24 16:01 amete

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Jan 19 '24 16:01 phsft-bot

Test Results

11 files 11 suites 2d 4h 52m 30s :stopwatch: 2 595 tests 2 593 :white_check_mark: 0 :zzz: 2 :x: 26 955 runs 26 953 :white_check_mark: 0 :zzz: 2 :x:

For more details on these failures, see this check.

Results for commit c9774121.

:recycle: This comment has been updated with latest results.

Jan 19 '24 22:01 github-actions[bot]

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Jan 30 '24 19:01 phsft-bot

Build failed on windows10/default. Running on null:C:\build\workspace\root-pullrequests-build See console output.

Errors:

[2024-01-30T20:03:46.363Z] C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\include\utility(244,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\ASTMerge.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:46.363Z] C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\include\functional(2740,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\MultiplexConsumer.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\llvm\include\llvm/Support/TrailingObjects.h(221,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\CreateInvocationFromCommandLine.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\include\clang/AST/DeclCXX.h(3539,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\InitPreprocessor.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\llvm\include\llvm/ADT/iterator.h(280,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\ChainedIncludesSource.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\include\xmemory(913,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\PrecompiledPreamble.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\include\clang/AST/Stmt.h(3397,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\DependencyFile.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\include\xpolymorphic_allocator.h(28,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\ModuleDependencyCollector.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\include\clang/AST/RecursiveASTVisitor.h(2719,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\ASTConsumers.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]
[2024-01-30T20:03:47.678Z] C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\llvm\include\llvm/ADT/ArrayRef.h(426,1): fatal error C1060: compiler is out of heap space (compiling source file C:\build\workspace\root-pullrequests-build\root\interpreter\llvm-project\clang\lib\Frontend\CompilerInstance.cpp) [C:\build\workspace\root-pullrequests-build\build\interpreter\llvm-project\llvm\tools\clang\lib\Frontend\obj.clangFrontend.vcxproj]

And 57 more

Jan 30 '24 20:01 phsft-bot

accept that we can't handle RNTuples that are not in the files root directory.

Should we open an issue to keep track of the places we will need to update when we want to support sub-directories?

Jan 30 '24 20:01 pcanal

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Feb 20 '24 19:02 phsft-bot

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Feb 20 '24 20:02 phsft-bot

Build failed on ROOT-ubuntu2204/nortcxxmod. Running on root-ubuntu-2204-3.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build See console output.

Failing tests:

Feb 20 '24 20:02 phsft-bot

Build failed on ROOT-ubuntu2004/python3. Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build See console output.

Failing tests:

Feb 20 '24 21:02 phsft-bot

Build failed on ROOT-performance-centos8-multicore/soversion. Running on olbdw-01.cern.ch:/data/sftnight/workspace/root-pullrequests-build See console output.

Failing tests:

projectroot.runtutorials.tutorial_rcanvas_rline.cxx

Feb 21 '24 00:02 phsft-bot

As a heads-up: I tried this PR to integrate RNTuple merging into TBufferMerger. It needs some change (for example RPageSourceFile needs to work with a TMemFile, ie we need a RRawFileTFile that wraps a TFile pointer), but then it mostly runs through. The only problem is that this code doesn't consider merging into a ntuple that already exists in the output file. As a result, all pages will be there but the meta data will only have the cluster(s) from the last merge. What we probably need is the RPageSinkFile for the destination to load an existing RNTuple anchor and append to it... If there are other approaches, I'm all ear but otherwise I don't think it should block this PR, just something to keep in mind for the future.

Feb 21 '24 14:02 hahnjo

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Feb 23 '24 14:02 phsft-bot

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Feb 23 '24 16:02 phsft-bot

Thanks a lot again @jblomer, @hahnjo. As we discussed at the meeting today, I adopted the suggestions, added a unit test that merges RNTuple through TFileMerger, and rebased the branched on top of master. I think this PR is good to go. Therefore, I'm marking it ready. Please let me know.

Feb 23 '24 16:02 amete

From my point of view, this is good to merge (we should probably squash the 9 commits into one while accepting). Thanks for wiring up the merging, further improvements can come in later PRs.

Feb 27 '24 07:02 hahnjo

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Feb 29 '24 16:02 phsft-bot

Starting build on ROOT-performance-centos8-multicore/soversion, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac12arm/cxx20, windows10/default How to customize builds

Feb 29 '24 17:02 phsft-bot

Thanks @jblomer. The last version should have fixed that. I also squashed the commits while at it. Please let me know.

Feb 29 '24 17:02 amete