cmssw
cmssw copied to clipboard
cms-sw
Segmentation violation when running `hltGetConfiguration` with option `-output full`
I was trying to answer the question from Andrea at: cms-sw/cmssw#37207 (comment) using the following script in CMSSW_15_0_2_patch1:
#!/bin/bash -ex
hltGetConfiguration /dev/CMSSW_15_0_0/GRun \
--globaltag 150X_dataRun3_HLT_v1 \
--data \
--unprescale \
--output full \
--max-events 100 \
--eras Run3_2024 --l1-emulator uGT --l1 L1Menu_Collisions2024_v1_3_0_xml \
--input /store/data/Run2024I/EphemeralHLTPhysics0/RAW/v1/000/386/593/00000/91a08676-199e-404c-9957-f72772ef1354.root \
> hltData.py
cmsRun hltData.py >& hltData.log
(using "full" output module, which according to important note in the SWGuideGlobalHLT section should probe the issue) and I got a segmentation fault:
Thread 1 (Thread 0x7f51ff429540 (LWP 597176) "cmsRun"):
#0 0x00007f51fef015bf in poll () from /lib64/libc.so.6
#1 0x00007f51f8c543b7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2 0x00007f51f8c545b4 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x00007f51e4de1d9f in clang::StringLiteralParser::init(llvm::ArrayRef<clang::Token>) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#5 0x00007f51e1f41605 in clang::Sema::ActOnStringLiteral(llvm::ArrayRef<clang::Token>, clang::Scope*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#6 0x0000000000000000 in ?? ()
using minimal and all I don't see the issue.
As per documentation:
--output all: default, all output modules are retained--output none: all output modules are removed--output minimal: all output modules are removed, a single one is added which just keeps trigger-related outputs (e.g. HLT results, L1T objects)--output full: all output modules are removed, a single one is added which saves everything
in the case that crashes, the PoolOutputModule is configured as:
# add a single "keep *" output
process.hltOutputFull = cms.OutputModule( "PoolOutputModule",
fileName = cms.untracked.string( "output.root" ),
fastCloning = cms.untracked.bool( False ),
dataset = cms.untracked.PSet(
dataTier = cms.untracked.string( 'RECO' ),
filterName = cms.untracked.string( '' )
),
outputCommands = cms.untracked.vstring( 'keep *' )
)
process.FullOutput = cms.EndPath( process.hltOutputFull )
process.schedule.append( process.FullOutput )
A new Issue was created by @mmusich.
@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
New categories assigned: hlt
@Martin-Grunewald,@mmusich you have been requested to review this Pull request/Issue and eventually sign? Thanks
Thread 1 (Thread 0x7f51ff429540 (LWP 597176) "cmsRun"): #0 0x00007f51fef015bf in poll () from /lib64/libc.so.6 #1 0x00007f51f8c543b7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so #2 0x00007f51f8c545b4 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 <signal handler called> #4 0x00007f51e4de1d9f in clang::StringLiteralParser::init(llvm::ArrayRef<clang::Token>) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so #5 0x00007f51e1f41605 in clang::Sema::ActOnStringLiteral(llvm::ArrayRef<clang::Token>, clang::Scope*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so #6 0x0000000000000000 in ?? ()
Was the stack trace cut? (i.e. is the above all the stack trace available, or is there more)
Was the stack trace cut? (i.e. is the above all the stack trace available, or is there more)
yes I can provide it in full. You can find it here.
Was the stack trace cut? (i.e. is the above all the stack trace available, or is there more)
yes I can provide it in full. You can find it here.
Thanks. From the bottom of the stack
#55 0x00007fd23d23514c in TCling::AutoParse(char const*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#56 0x00007fd23d27c7ce in TClingCallbacks::LookupObject(clang::TagDecl*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#57 0x00007fd23d30c35c in cling::MultiplexInterpreterCallbacks::LookupObject(clang::TagDecl*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#58 0x00007fd23dfefd20 in clang::MultiplexExternalSemaSource::CompleteType(clang::TagDecl*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#59 0x00007fd23e91890b in clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::CompleteTypeKind, clang::Sema::TypeDiagnoser*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#60 0x00007fd23e918fa5 in clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::CompleteTypeKind, clang::Sema::TypeDiagnoser&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#61 0x00007fd23e04a3a8 in clang::Sema::RequireCompleteDeclContext(clang::CXXScopeSpec&, clang::DeclContext*&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#62 0x00007fd23e737ad8 in clang::Sema::LookupTemplateName(clang::LookupResult&, clang::Scope*, clang::CXXScopeSpec&, clang::QualType, bool, bool&, clang::Sema::RequiredTemplateKind, clang::Sema::AssumedTemplateKind*, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#63 0x00007fd23e7393aa in clang::Sema::isTemplateName(clang::Scope*, clang::CXXScopeSpec&, bool, clang::UnqualifiedId const&, clang::OpaquePtr<clang::QualType>, bool, clang::OpaquePtr<clang::TemplateName>&, bool&, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#64 0x00007fd23ddea208 in clang::Parser::ParseOptionalCXXScopeSpecifier(clang::CXXScopeSpec&, clang::OpaquePtr<clang::QualType>, bool, bool, bool*, bool, clang::IdentifierInfo**, bool, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#65 0x00007fd23de747a3 in clang::Parser::TryAnnotateCXXScopeToken(bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#66 0x00007fd23d31be6e in cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#67 0x00007fd23d2382d2 in TCling::CheckClassInfo(char const*, bool, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#68 0x00007fd25c118c22 in TClass::Init(char const*, short, std::type_info const*, TVirtualIsAProxy*, char const*, char const*, int, int, ClassInfo_t*, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#69 0x00007fd25c1199f8 in TClass::TClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#70 0x00007fd25c119afb in ROOT::CreateClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#71 0x00007fd25c12b002 in ROOT::TGenericClassInfo::GetClass() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#72 0x00007fd25c10db3a in TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#73 0x00007fd25c145e65 in TStreamerElement::GetClassPointer() const () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#74 0x00007fd250ddc1e0 in TStreamerInfo::Compile() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libRIO.so
#75 0x00007fd250daae55 in TStreamerInfo::Build(bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libRIO.so
#76 0x00007fd25c104b7c in TClass::GetStreamerInfoImpl(int, bool) const () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#77 0x00007fd25c104d1e in TClass::GetStreamerInfo(int, bool) const () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#78 0x00007fd25c148390 in TStreamerBase::InitStreaming(bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#79 0x00007fd25c148f8b in TStreamerBase::TStreamerBase(char const*, char const*, int, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#80 0x00007fd250daafb9 in TStreamerInfo::Build(bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libRIO.so
#81 0x00007fd25c104b7c in TClass::GetStreamerInfoImpl(int, bool) const () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#82 0x00007fd25c104d1e in TClass::GetStreamerInfo(int, bool) const () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#83 0x00007fd250d8d30c in TStreamerInfo::ForceWriteInfo(TFile*, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libRIO.so
#84 0x00007fd254dd84de in TTree::BuildStreamerInfo(TClass*, void*, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libTree.so
#85 0x00007fd254deb202 in TTree::BronchExec(char const*, char const*, void*, bool, int, int) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libTree.so
#86 0x00007fd254dd67e1 in TTree::Branch(char const*, char const*, void*, int, int) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libTree.so
#87 0x00007fd1b6e6951b in edm::RootOutputTree::addBranch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*&, int, int, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/libIOPoolOutput.so
#88 0x00007fd1b6e6a6d9 in edm::RootOutputFile::RootOutputFile(edm::PoolOutputModule*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/libIOPoolOutput.so
#89 0x00007fd1b6e53507 in edm::PoolOutputModule::reallyOpenFile() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/libIOPoolOutput.so
#90 0x00007fd1b6e53661 in virtual thunk to edm::PoolOutputModule::openFile(edm::FileBlock const&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/libIOPoolOutput.so
#91 0x00007fd25c6046f8 in edm::Schedule::openOutputFiles(edm::FileBlock&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/libFWCoreFramework.so
#92 0x00007fd25c551fcd in edm::EventProcessor::openOutputFiles() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/libFWCoreFramework.so
#93 0x00007fd25c555fde in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_0_2/lib/el9_amd64_gcc12/libFWCoreFramework.so
i.e. something leads to auto-parsing of headers. I wonder if there could any connection to https://github.com/cms-sw/cmssw/issues/47470#issuecomment-2698410582 / https://github.com/root-project/root/issues/17992 (maybe not, there is no indication of library load here)
We'd need to dig in what TBranch exactly is being created, and for what class the TClass is being created.
We'd need to dig in what TBranch exactly is being created
The branch in question is SiPixelClustersHost_hltSiPixelClustersSoA__HLTX. for the (wrapped) class name edm::Wrapper<SiPixelClustersHost>
and for what class the
TClassis being created.
The TClass::GetClass() call before the crash was for SiPixelClustersLayout<128,false>::ViewTemplateFreeParams<128,false,true,true> (after many other TClass::GetClass() calls).
Adding a dictionary for SiPixelClustersLayout<128,false>::ViewTemplateFreeParams<128,false,true,true> didn't help.
There seems to be some connection to https://github.com/root-project/root/issues/17992, because repeating the test in CMSSW_15_1_X_2025-03-25-2300 (where the ROOT has been updated to include a fix https://github.com/cms-sw/cmsdist/pull/9757) leads to an exception instead of a crash
----- Begin Fatal Exception 26-Mar-2025 17:23:59 CET-----------------------
An exception of category 'FatalRootError' occurred while
[0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Additional Info:
[a] Fatal Root Error: @SUB=TInterpreter::AutoParse
Error parsing payload code for class SiPixelClustersLayout with content:
#line 1 "DataFormatsSiPixelClusterSoA_xr dictionary payload"
#ifndef ALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT
#define ALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT 128
#endif
<cut>
#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#ifndef DataFormats_SiPixelClusterSoA_src_classes_h
#define DataFormats_SiPixelClusterSoA_src_classes_h
#include "DataFormats/Common/interface/Wrapper.h"
#include "DataFormats/SiPixelClusterSoA/interface/SiPixelClustersHost.h"
#endif // DataFormats_SiPixelClusterSoA_src_classes_h
#undef _BACKWARD_BACKWARD_WARNING_H
----- End Fatal Exception -------------------------------------------------
As a cross check (as it was hinted in the description) I reran the job keeping only the SiPixelClustersHost_hltSiPixelClustersSoA__HLTX with
process.hltOutputFull.outputCommands = ["drop *", "keep SiPixelClustersHost_hltSiPixelClustersSoA__HLTX"]
, and that works. So there is indeed some connection to other data types that are being kept (for which TClass::GetClass() is being called?).
@pcanal Would you have any insight?
Tagging also @cms-sw/heterogeneous-l2 given the connection to the SoA.
(in earlier tests the job was configured to use 4 threads, what I report here is with 1 thread)
repeating the test in
CMSSW_15_1_X_2025-03-25-2300(where the ROOT has been updated to include a fix cms-sw/cmsdist#9757) leads to an exception instead of a crash
Well, repeating the test a few more times I see that some times it segfaults and some times it results an exception. Occasionally the stack trace shows symptoms of memory corruption
Thread 1 (Thread 0x7f2a42c7d580 (LWP 220962) "cmsRun"):
#0 0x00007f2a4430aac1 in poll () from /lib64/libc.so.6
#1 0x00007f2a3a451d57 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2 0x00007f2a3a451f54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x00007f2a257b9f5f in clang::StringLiteralParser::init(llvm::ArrayRef<clang::Token>) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/external/el8_amd64_gcc12/lib/libCling.so
#5 0x00007f2a22911e35 in clang::Sema::ActOnStringLiteral(llvm::ArrayRef<clang::Token>, clang::Scope*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/external/el8_amd64_gcc12/lib/libCling.so
#6 0x0000000100007f29 in ?? ()
#7 0xa6bdce9000000004 in ?? ()
#8 0x0000000000007f29 in ?? ()
#9 0x0000001500000000 in ?? ()
#10 0x014e961f00007f2a in ?? ()
#11 0x34a6919000000018 in ?? ()
#12 0x0000000500007f2a in ?? ()
#13 0x014e963700007f2a in ?? ()
#14 0x0000000000000001 in ?? ()
#15 0x0000004100000000 in ?? ()
#16 0x014e963900007f2a in ?? ()
#17 0x0000000000000001 in ?? ()
#18 0x0000000000000000 in ?? ()
Ok, after switching to 1 thread, I observe the exception behavior also in CMSSW_15_0_2_patch1 (i.e. some times the job segfaults and some times it results in an exception).
Given the somewhat random behavior, I'm checking if valgrind would give any insight.
New categories assigned: core
@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks
Given the somewhat random behavior, I'm checking if valgrind would give any insight.
I ran valgrind on the CMSSW_15_1_X_2025-03-25-2300 setup, and there are 36 Invalid read of size 1 warnings along
==222124== Invalid read of size 1
==222124== at 0x20CB60FD: clang::NumericLiteralParser::NumericLiteralParser(llvm::StringRef, clang::SourceLocation, clang::SourceManager const&, clang::LangOptions const&, clang::TargetInfo const&, clang::DiagnosticsEngine&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1DDED951: clang::Sema::ActOnNumericConstant(clang::Token const&, clang::Scope*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D8A582F: clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, bool&, clang::Parser::TypeCastState, bool, bool*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D8A82B8: clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, clang::Parser::TypeCastState, bool, bool*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D8AB217: clang::Parser::ParseAssignmentExpression(clang::Parser::TypeCastState) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D888DE0: clang::Parser::ParseCXXMemberInitializer(clang::Decl*, bool, clang::SourceLocation&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D89BA2F: clang::Parser::ParseCXXClassMemberDeclaration(clang::AccessSpecifier, clang::ParsedAttributes&, clang::Parser::ParsedTemplateInfo const&, clang::ParsingDeclRAIIObject*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D89CBE2: clang::Parser::ParseCXXClassMemberDeclarationWithPragmas(clang::AccessSpecifier&, clang::ParsedAttributes&, clang::TypeSpecifierType, clang::Decl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D89CFCF: clang::Parser::ParseCXXMemberSpecification(clang::SourceLocation, clang::SourceLocation, clang::ParsedAttributes&, unsigned int, clang::Decl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D89F43A: clang::Parser::ParseClassSpecifier(clang::tok::TokenKind, clang::SourceLocation, clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, bool, clang::Parser::DeclSpecContext, clang::ParsedAttributes&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D86F373: clang::Parser::ParseDeclarationSpecifiers(clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, clang::Parser::DeclSpecContext, clang::Parser::LateParsedAttrList*, clang::ImplicitTypenameContext) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D935922: clang::Parser::ParseSingleDeclarationAfterTemplate(clang::DeclaratorContext, clang::Parser::ParsedTemplateInfo const&, clang::ParsingDeclRAIIObject&, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D93BC8C: clang::Parser::ParseTemplateDeclarationOrSpecialization(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D93BEB6: clang::Parser::ParseDeclarationStartingWithTemplate(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D87DB4B: clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D951413: clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D952C2A: clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CE70CC8: cling::IncrementalParser::ParseInternal(llvm::StringRef) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CE71CBB: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CDDA201: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== Address 0xa310262f is 35,855 bytes inside a block of size 57,544 free'd
==222124== at 0x403F50B: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/external/valgrind/3.24.0-f85a1303334507f502ed8242a93c05bd/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==222124== by 0x20DBD9A7: clang::SourceManager::invalidateCache(clang::FileID) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CE30F5D: cling::DeclUnloader::~DeclUnloader() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CDF9527: cling::TransactionUnloader::RevertTransaction(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CDD8972: cling::Interpreter::unload(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CE6E967: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CDD7959: cling::Interpreter::PushTransactionRAII::~PushTransactionRAII() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CDF130C: cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CD65183: TClingClassInfo::TClingClassInfo(cling::Interpreter*, char const*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CCF19DC: TCling::SetClassInfo(TClass*, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x50F8DFF: TClass::LoadClassInfo() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x50F90D7: TClass::GetListOfBases() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x50F91F8: TClass::GetBaseClass(TClass const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x50F92C2: TClass::InheritsFrom(TClass const*) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x50FC58D: TClass::Property() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x50FC3E9: TClass::IsForeign() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x5109E77: TClass::PostLoadCheck() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x5100DEB: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x51391F4: TStreamerElement::GetClassPointer() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==222124== by 0x1495A41F: TStreamerInfo::Compile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==222124== Block was alloc'd at
==222124== at 0x403C9DB: operator new(unsigned long, std::nothrow_t const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/external/valgrind/3.24.0-f85a1303334507f502ed8242a93c05bd/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==222124== by 0x211413A0: llvm::WritableMemoryBuffer::getNewUninitMemBuffer(unsigned long, llvm::Twine const&, std::optional<llvm::Align>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x211416F9: llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> > > getOpenFileImpl<llvm::MemoryBuffer>(int, llvm::Twine const&, unsigned long, unsigned long, long, bool, bool, std::optional<llvm::Align>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x21142604: llvm::MemoryBuffer::getOpenFile(int, llvm::Twine const&, unsigned long, bool, bool, std::optional<llvm::Align>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x2119EA99: (anonymous namespace)::RealFile::getBuffer(llvm::Twine const&, long, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20D730B4: clang::FileManager::getBufferForFile(clang::FileEntry const*, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20DB55CB: clang::SrcMgr::ContentCache::getBufferOrNone(clang::DiagnosticsEngine&, clang::FileManager&, clang::SourceLocation) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20CF91A6: clang::Preprocessor::EnterSourceFile(clang::FileID, clang::detail::SearchDirIteratorImpl<true>, clang::SourceLocation, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20CEF106: clang::Preprocessor::HandleHeaderIncludeOrImport(clang::SourceLocation, clang::Token&, clang::Token&, clang::SourceLocation, clang::detail::SearchDirIteratorImpl<true>, clang::FileEntry const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20CEF8C5: clang::Preprocessor::HandleIncludeDirective(clang::SourceLocation, clang::Token&, clang::detail::SearchDirIteratorImpl<true>, clang::FileEntry const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20CF068D: clang::Preprocessor::HandleDirective(clang::Token&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20CAD3A4: clang::Lexer::LexTokenInternal(clang::Token&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x20D2A5EE: clang::Preprocessor::Lex(clang::Token&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D86189A: clang::Parser::ConsumeBrace() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D868D2B: clang::BalancedDelimiterTracker::consumeClose() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D898710: clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D87DC1D: clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D951413: clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1D952C2A: clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1CE70CC8: cling::IncrementalParser::ParseInternal(llvm::StringRef) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
followed by
==222124== Invalid write of size 1
==222124== at 0x20CBEF5F: clang::StringLiteralParser::init(llvm::ArrayRef<clang::Token>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1DE16E34: clang::Sema::ActOnStringLiteral(llvm::ArrayRef<clang::Token>, clang::Scope*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==222124== by 0x1DFCD43FFFFFFFFF: ???
==222124== by 0x202001FFFFFFFFFF: ???
==222124== by 0x1: ???
==222124== by 0x12FFFFFFFF: ???
==222124== Address 0x1300001345 is not stack'd, malloc'd or (recently) free'd
==222124==
A fatal system signal has occurred: segmentation violation
I attached the full valgrind log valgrind.txt.
It seems like the following outputCommands is sufficient to trigger the problem
process.hltOutputFull.outputCommands = [
"drop *",
"keep 128falsehcalHcalRecHitSoALayoutPortableHostCollection_*_*_*",
"keep 128falserecoPFRecHitSoALayoutPortableHostCollection_*_*_*",
"keep SiPixelClustersHost_hltSiPixelClustersSoA__HLTX",
Dropping any one of the keep statements seem to make the job succeed.
If I read the SoATemplate macro code correctly, the type SiPixelClustersLayout<128,false>::View is an alias for SiPixelClustersLayout<128,false>::ViewTemplateFreeParams<128,false,true,true>, and the dictionary for the dictionary for SiPixelClustersLayout<128,false>::View is requested in
https://github.com/cms-sw/cmssw/blob/7a8aff4ce3c54f73b53e996aeb0c6369cd226d7b/DataFormats/SiPixelClusterSoA/src/classes_def.xml#L3
I don't think we should have a dictionary for the View - either we shouldn't mention it in the XML file or it should be transient.
Anyway, removing it doesn't seem to make a difference.
Given the somewhat random behavior, I'm checking if valgrind would give any insight.
I ran valgrind on the
CMSSW_15_1_X_2025-03-25-2300setup
I re-ran the valgrind with higher --num-callers and with some extra instrumentation. In this case the job terminated with the exception. There were total of 25 Invalid reads, of which 20 were of size 1, 4 were of size 2, and 1 was of size 8. The warnings are concentrated on the edm::RootOutputTree::addBranch() call for SiPixelClustersHost_hltSiPixelClustersSoA__HLTX., and within there, to the TClass::GetClass() call for SiPixelClustersLayout<128,false>::ViewTemplateFreeParams<128,false,true,true>.
Below is the first warning, and the full log is here valgrind2.txt
==235527== Invalid read of size 1
==235527== at 0x20CB60FD: clang::NumericLiteralParser::NumericLiteralParser(llvm::StringRef, clang::SourceLocation, clang::SourceManager const&, clang::LangOptions const&, clang::TargetInfo const&, clang::DiagnosticsEngine&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1DDED951: clang::Sema::ActOnNumericConstant(clang::Token const&, clang::Scope*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D8A582F: clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, bool&, clang::Parser::TypeCastState, bool, bool*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D8A82B8: clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, clang::Parser::TypeCastState, bool, bool*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D8AB217: clang::Parser::ParseAssignmentExpression(clang::Parser::TypeCastState) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D888DE0: clang::Parser::ParseCXXMemberInitializer(clang::Decl*, bool, clang::SourceLocation&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D89BA2F: clang::Parser::ParseCXXClassMemberDeclaration(clang::AccessSpecifier, clang::ParsedAttributes&, clang::Parser::ParsedTemplateInfo const&, clang::ParsingDeclRAIIObject*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D89CBE2: clang::Parser::ParseCXXClassMemberDeclarationWithPragmas(clang::AccessSpecifier&, clang::ParsedAttributes&, clang::TypeSpecifierType, clang::Decl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D89CFCF: clang::Parser::ParseCXXMemberSpecification(clang::SourceLocation, clang::SourceLocation, clang::ParsedAttributes&, unsigned int, clang::Decl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D89F43A: clang::Parser::ParseClassSpecifier(clang::tok::TokenKind, clang::SourceLocation, clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, bool, clang::Parser::DeclSpecContext, clang::ParsedAttributes&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D86F373: clang::Parser::ParseDeclarationSpecifiers(clang::DeclSpec&, clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, clang::Parser::DeclSpecContext, clang::Parser::LateParsedAttrList*, clang::ImplicitTypenameContext) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D935922: clang::Parser::ParseSingleDeclarationAfterTemplate(clang::DeclaratorContext, clang::Parser::ParsedTemplateInfo const&, clang::ParsingDeclRAIIObject&, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D93BC8C: clang::Parser::ParseTemplateDeclarationOrSpecialization(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D93BEB6: clang::Parser::ParseDeclarationStartingWithTemplate(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::AccessSpecifier) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D87DB4B: clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D951413: clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D952C2A: clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CE70CC8: cling::IncrementalParser::ParseInternal(llvm::StringRef) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CE71CBB: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDDA201: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CCFC674: ExecAutoParse(char const*, bool, cling::Interpreter*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CCFEA32: TCling::AutoParseImplRecurse(char const*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CD098EB: TCling::AutoParse(char const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CD50F6D: TClingCallbacks::LookupObject(clang::TagDecl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDE113B: cling::MultiplexInterpreterCallbacks::LookupObject(clang::TagDecl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1DAC543F: clang::MultiplexExternalSemaSource::CompleteType(clang::TagDecl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E3EE13A: clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::CompleteTypeKind, clang::Sema::TypeDiagnoser*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E3EE7D4: clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::CompleteTypeKind, clang::Sema::TypeDiagnoser&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1DB1FAC7: clang::Sema::RequireCompleteDeclContext(clang::CXXScopeSpec&, clang::DeclContext*&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E20D307: clang::Sema::LookupTemplateName(clang::LookupResult&, clang::Scope*, clang::CXXScopeSpec&, clang::QualType, bool, bool&, clang::Sema::RequiredTemplateKind, clang::Sema::AssumedTemplateKind*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E20EBD9: clang::Sema::isTemplateName(clang::Scope*, clang::CXXScopeSpec&, bool, clang::UnqualifiedId const&, clang::OpaquePtr<clang::QualType>, bool, clang::OpaquePtr<clang::TemplateName>&, bool&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D8BED07: clang::Parser::ParseOptionalCXXScopeSpecifier(clang::CXXScopeSpec&, clang::OpaquePtr<clang::QualType>, bool, bool, bool*, bool, clang::IdentifierInfo**, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D9492A2: clang::Parser::TryAnnotateCXXScopeToken(bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDF0C3D: cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CD0CA71: TCling::CheckClassInfo(char const*, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x510BEF1: TClass::Init(char const*, short, std::type_info const*, TVirtualIsAProxy*, char const*, char const*, int, int, ClassInfo_t*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x510D407: TClass::TClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x510D96A: ROOT::CreateClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x511E391: ROOT::TGenericClassInfo::GetClass() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x5100DD1: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x4052258: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libPerfToolsHack.so)
==235527== by 0x51391F4: TStreamerElement::GetClassPointer() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x14EDD41F: TStreamerInfo::Compile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x14EAC094: TStreamerInfo::Build(bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x50F7C2B: TClass::GetStreamerInfoImpl(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50F7DCD: TClass::GetStreamerInfo(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x513B71F: TStreamerBase::InitStreaming(bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x513C31A: TStreamerBase::TStreamerBase(char const*, char const*, int, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x14EAC1F8: TStreamerInfo::Build(bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x50F7C2B: TClass::GetStreamerInfoImpl(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50F7DCD: TClass::GetStreamerInfo(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x14E8E4EB: TStreamerInfo::ForceWriteInfo(TFile*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x12BEC5BD: TTree::BuildStreamerInfo(TClass*, void*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x12BFF2E1: TTree::BronchExec(char const*, char const*, void*, bool, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x12BEA8C0: TTree::Branch(char const*, char const*, void*, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x9E27B568: edm::RootOutputTree::addBranch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*&, int, int, bool) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E27C8D8: edm::RootOutputFile::RootOutputFile(edm::PoolOutputModule*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E265A86: edm::PoolOutputModule::reallyOpenFile() (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E265BE0: virtual thunk to edm::PoolOutputModule::openFile(edm::FileBlock const&) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x4C42427: edm::Schedule::openOutputFiles(edm::FileBlock&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x4B86F3C: edm::EventProcessor::openOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x4B8D86D: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x408555: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== by 0x5583F70: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:821)
==235527== by 0x40A282: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== by 0x4051B7: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== Address 0xa5499acf is 35,855 bytes inside a block of size 57,544 free'd
==235527== at 0x403F50B: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/external/valgrind/3.24.0-f85a1303334507f502ed8242a93c05bd/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==235527== by 0x20DBD9A7: clang::SourceManager::invalidateCache(clang::FileID) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CE30F5D: cling::DeclUnloader::~DeclUnloader() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDF9527: cling::TransactionUnloader::RevertTransaction(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDD8972: cling::Interpreter::unload(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CE6E967: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDD7959: cling::Interpreter::PushTransactionRAII::~PushTransactionRAII() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDF130C: cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CD65183: TClingClassInfo::TClingClassInfo(cling::Interpreter*, char const*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CCF19DC: TCling::SetClassInfo(TClass*, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x50F8DFF: TClass::LoadClassInfo() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50F90D7: TClass::GetListOfBases() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50F91F8: TClass::GetBaseClass(TClass const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50F92C2: TClass::InheritsFrom(TClass const*) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50FC58D: TClass::Property() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50FC3E9: TClass::IsForeign() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x5109E77: TClass::PostLoadCheck() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x5100DEB: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x4052258: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libPerfToolsHack.so)
==235527== by 0x51391F4: TStreamerElement::GetClassPointer() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x14EDD41F: TStreamerInfo::Compile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x14EAC094: TStreamerInfo::Build(bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x50F7C2B: TClass::GetStreamerInfoImpl(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50F7DCD: TClass::GetStreamerInfo(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x14E8E4EB: TStreamerInfo::ForceWriteInfo(TFile*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x12BEC5BD: TTree::BuildStreamerInfo(TClass*, void*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x12BFF2E1: TTree::BronchExec(char const*, char const*, void*, bool, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x12BEA8C0: TTree::Branch(char const*, char const*, void*, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x9E27B568: edm::RootOutputTree::addBranch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*&, int, int, bool) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E27C8D8: edm::RootOutputFile::RootOutputFile(edm::PoolOutputModule*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E265A86: edm::PoolOutputModule::reallyOpenFile() (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E265BE0: virtual thunk to edm::PoolOutputModule::openFile(edm::FileBlock const&) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x4C42427: edm::Schedule::openOutputFiles(edm::FileBlock&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x4B86F3C: edm::EventProcessor::openOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x4B8D86D: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x408555: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== by 0x5583F70: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:821)
==235527== by 0x40A282: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== by 0x4051B7: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== Block was alloc'd at
==235527== at 0x403C9DB: operator new(unsigned long, std::nothrow_t const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/external/valgrind/3.24.0-f85a1303334507f502ed8242a93c05bd/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==235527== by 0x211413A0: llvm::WritableMemoryBuffer::getNewUninitMemBuffer(unsigned long, llvm::Twine const&, std::optional<llvm::Align>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x211416F9: llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> > > getOpenFileImpl<llvm::MemoryBuffer>(int, llvm::Twine const&, unsigned long, unsigned long, long, bool, bool, std::optional<llvm::Align>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x21142604: llvm::MemoryBuffer::getOpenFile(int, llvm::Twine const&, unsigned long, bool, bool, std::optional<llvm::Align>) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x2119EA99: (anonymous namespace)::RealFile::getBuffer(llvm::Twine const&, long, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20D730B4: clang::FileManager::getBufferForFile(clang::FileEntry const*, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20DB55CB: clang::SrcMgr::ContentCache::getBufferOrNone(clang::DiagnosticsEngine&, clang::FileManager&, clang::SourceLocation) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20CF91A6: clang::Preprocessor::EnterSourceFile(clang::FileID, clang::detail::SearchDirIteratorImpl<true>, clang::SourceLocation, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20CEF106: clang::Preprocessor::HandleHeaderIncludeOrImport(clang::SourceLocation, clang::Token&, clang::Token&, clang::SourceLocation, clang::detail::SearchDirIteratorImpl<true>, clang::FileEntry const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20CEF8C5: clang::Preprocessor::HandleIncludeDirective(clang::SourceLocation, clang::Token&, clang::detail::SearchDirIteratorImpl<true>, clang::FileEntry const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20CF068D: clang::Preprocessor::HandleDirective(clang::Token&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20CAD3A4: clang::Lexer::LexTokenInternal(clang::Token&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x20D2A5EE: clang::Preprocessor::Lex(clang::Token&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D86189A: clang::Parser::ConsumeBrace() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D868D2B: clang::BalancedDelimiterTracker::consumeClose() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D898710: clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D87DC1D: clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D951413: clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D952C2A: clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CE70CC8: cling::IncrementalParser::ParseInternal(llvm::StringRef) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CE71CBB: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDDA201: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CCFC674: ExecAutoParse(char const*, bool, cling::Interpreter*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CCFEA32: TCling::AutoParseImplRecurse(char const*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CD098EB: TCling::AutoParse(char const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CD50F6D: TClingCallbacks::LookupObject(clang::TagDecl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDE113B: cling::MultiplexInterpreterCallbacks::LookupObject(clang::TagDecl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1DAC543F: clang::MultiplexExternalSemaSource::CompleteType(clang::TagDecl*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E3EE13A: clang::Sema::RequireCompleteTypeImpl(clang::SourceLocation, clang::QualType, clang::Sema::CompleteTypeKind, clang::Sema::TypeDiagnoser*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E3EE7D4: clang::Sema::RequireCompleteType(clang::SourceLocation, clang::QualType, clang::Sema::CompleteTypeKind, clang::Sema::TypeDiagnoser&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1DB1FAC7: clang::Sema::RequireCompleteDeclContext(clang::CXXScopeSpec&, clang::DeclContext*&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E20D307: clang::Sema::LookupTemplateName(clang::LookupResult&, clang::Scope*, clang::CXXScopeSpec&, clang::QualType, bool, bool&, clang::Sema::RequiredTemplateKind, clang::Sema::AssumedTemplateKind*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1E20EBD9: clang::Sema::isTemplateName(clang::Scope*, clang::CXXScopeSpec&, bool, clang::UnqualifiedId const&, clang::OpaquePtr<clang::QualType>, bool, clang::OpaquePtr<clang::TemplateName>&, bool&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D8BED07: clang::Parser::ParseOptionalCXXScopeSpecifier(clang::CXXScopeSpec&, clang::OpaquePtr<clang::QualType>, bool, bool, bool*, bool, clang::IdentifierInfo**, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1D9492A2: clang::Parser::TryAnnotateCXXScopeToken(bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CDF0C3D: cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x1CD0CA71: TCling::CheckClassInfo(char const*, bool, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCling.so)
==235527== by 0x510BEF1: TClass::Init(char const*, short, std::type_info const*, TVirtualIsAProxy*, char const*, char const*, int, int, ClassInfo_t*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x510D407: TClass::TClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x510D96A: ROOT::CreateClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x511E391: ROOT::TGenericClassInfo::GetClass() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x5100DD1: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x4052258: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libPerfToolsHack.so)
==235527== by 0x51391F4: TStreamerElement::GetClassPointer() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x14EDD41F: TStreamerInfo::Compile() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x14EAC094: TStreamerInfo::Build(bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x50F7C2B: TClass::GetStreamerInfoImpl(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x50F7DCD: TClass::GetStreamerInfo(int, bool) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libCore.so)
==235527== by 0x14E8E4EB: TStreamerInfo::ForceWriteInfo(TFile*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libRIO.so)
==235527== by 0x12BEC5BD: TTree::BuildStreamerInfo(TClass*, void*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x12BFF2E1: TTree::BronchExec(char const*, char const*, void*, bool, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x12BEA8C0: TTree::Branch(char const*, char const*, void*, int, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/lcg/root/6.32.11-fbe70b72bc88f3b2c229ad240e953f56/lib/libTree.so)
==235527== by 0x9E27B568: edm::RootOutputTree::addBranch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*&, int, int, bool) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E27C8D8: edm::RootOutputFile::RootOutputFile(edm::PoolOutputModule*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E265A86: edm::PoolOutputModule::reallyOpenFile() (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x9E265BE0: virtual thunk to edm::PoolOutputModule::openFile(edm::FileBlock const&) (in /build/mkortela/debug/issue47697/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libIOPoolOutput.so)
==235527== by 0x4C42427: edm::Schedule::openOutputFiles(edm::FileBlock&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x4B86F3C: edm::EventProcessor::openOutputFiles() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x4B8D86D: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so)
==235527== by 0x408555: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== by 0x5583F70: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:821)
==235527== by 0x40A282: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
==235527== by 0x4051B7: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02882/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-03-25-2300/bin/el8_amd64_gcc12/cmsRun)
Any error in Clang/LLVM after something like:
An exception of category 'FatalRootError' occurred while
[0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Additional Info:
[a] Fatal Root Error: @SUB=TInterpreter::AutoParse
Error parsing payload code for class SiPixelClustersLayout with content:
is most likely a 'bug' in the Cling/ErrorRecovery code (i.e. unloading some memory but 'forgetting' to deregister some entities and/or intentionally keeping some entities but forgetting to prevent the deletion of the memory).
If the 1st parameter on stack 69, 67 and 66 is "SiPixelClustersLayout<128,false>::ViewTemplateFreeParams<128,false,true,true>"
#66 0x00007fd23d31be6e in cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#67 0x00007fd23d2382d2 in TCling::CheckClassInfo(char const*, bool, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCling.so
#68 0x00007fd25c118c22 in TClass::Init(char const*, short, std::type_info const*, TVirtualIsAProxy*, char const*, char const*, int, int, ClassInfo_t*, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
#69 0x00007fd25c1199f8 in TClass::TClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_0_2_patch1/external/el9_amd64_gcc12/lib/libCore.so
then
Adding a dictionary for SiPixelClustersLayout<128,false>::ViewTemplateFreeParams<128,false,true,true> didn't help.
Is very surprising. I would need a reproducer (with debug build of ROOT)
(with debug build of ROOT)
Building now in https://github.com/cms-sw/cmsdist/pull/9761.
The problem can be reproduced already with
import FWCore.ParameterSet.Config as cms
process = cms.Process( "HLTX")
process.load("Configuration.StandardSequences.Accelerators_cff")
process.source = cms.Source("EmptySource")
process.maxEvents.input = 1
process.hltOutputFull = cms.OutputModule( "PoolOutputModule",
fileName = cms.untracked.string( "output.root" ),
outputCommands = cms.untracked.vstring(
"drop *",
"keep 128falsehcalHcalRecHitSoALayoutPortableHostCollection_*_*_*",
"keep 128falserecoPFRecHitSoALayoutPortableHostCollection_*_*_*",
)
)
process.FullOutput = cms.EndPath( process.hltOutputFull )
process.hltHbheRecoSoASerialSync = cms.EDProducer( "alpaka_serial_sync::HBHERecHitProducerPortable")
process.hltParticleFlowRecHitHBHESoASerialSync = cms.EDProducer( "alpaka_serial_sync::PFRecHitSoAProducerHCAL")
process.p = cms.Path(
process.hltHbheRecoSoASerialSync +
process.hltParticleFlowRecHitHBHESoASerialSync
)
without any input file (normally this job would fail with a ProductNotFound exception). Notably the SiPixelClustersHost_hltSiPixelClustersSoA_*_* was not needed here to reproduce the crash.
In this case the job crashes when calling RootOutputTree::addBranch() for branch 128falserecoPFRecHitSoALayoutPortableHostCollection_hltParticleFlowRecHitHBHESoASerialSync__HLTX that corresponds to edm::Wrapper<PortableHostCollection<reco::PFRecHitSoALayout<128,false> > > type name. The crash occurred under TClass::GetClass() call for reco::PFRecHitSoALayout<128,false>::ViewTemplateFreeParams<128,false,true,true> (which is a similar nested type as in the earlier case, and for which the dictionary is requested in
https://github.com/cms-sw/cmssw/blob/7035c70e3a533533a7f8d600ff29f23579ca6add/DataFormats/ParticleFlowReco/src/classes_serial_def.xml#L10
)
@pcanal CMSSW with a ROOT debug build is available via
/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/45251/install.sh
The configuration above reproduces the problem without needing an input file.
The underlying problem is a confusion in the normalized name where ROOT is using sometimes reco::PFRecHitSoALayout<128UL,false>::ViewTemplateFreeParams<128,false,true,true> and sometimes reco::PFRecHitSoALayout<128,false>::ViewTemplateFreeParams<128,false,true,true> (respectively used in the rootpcm and in the list of base classes of a derived class ; the mismatch leads to needing to ask the interpreter for more information rather than using the pre-calculated information).
I am looking into how to best resolve this.
Ah, took me a few seconds to spot the difference... so it's 128UL vs 128 ?
Is it something internal to ROOT, or something we are using inconsistently on our side ?
The template parameter is of type std::size_t, so I guess the correct one should be 128UL.
But then the correct type should actually be reco::PFRecHitSoALayout<128UL,false>::ViewTemplateFreeParams<128UL,false,true,true> (note also the second 128UL).