cmssw icon indicating copy to clipboard operation
cmssw copied to clipboard
verified publisher icon cms-sw

"`validateJR`" object comparisons broken for step2 (HLT) in PR tests

Open mmusich opened this issue 2 weeks ago • 35 comments

Title says it all. See for example a log here. I put here a full log for convenience:

Start making plots for Events with 10 events and refEvents with 10 events ==> check  10
Branch edmErrorSummaryEntrys_logErrorHarvester__HLT.obj is not found in either of the inputs. Skip.
plotting variable: [email protected]()
plotting variable: [email protected]()
plotting variable: [email protected]()
plotting variable: [email protected]()
plotting variable: [email protected]()
plotting variable: [email protected]()
plotting variable: [email protected]()
plotting variable: HcalUpgradeDataFramesSorted_simHcalUnsuppressedDigis_HBHEUpgradeDigiCollection_HLT.obj.obj@.size()
In file included from DataFormatsEcalDigi_xr dictionary payload:76:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiHostCollection.h:5:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiSoA.h:6:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:11:
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:442:19: error: static assertion failed due to requirement '!sizeof(Eigen::Matrix<float, 5, 1, 0, 5, 1>)': Eigen/Core should be pre-included before the SoA headers to enable support for Eigen columns.
    static_assert(!sizeof(C),
                  ^~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:792:38: note: in instantiation of template class 'cms::soa::SoAValue<cms::soa::SoAColumnType::eigen, Eigen::Matrix<float, 5, 1, 0, 5, 1>, 0, true>' requested here
    using ParamReturnType = typename SoAValue<SoAColumnType::eigen, T, alignment, restrictQualify>::CMapType;
                                     ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:813:24: note: in instantiation of template class 'cms::soa::SoAColumnAccessorsImpl<Eigen::Matrix<float, 5, 1, 0, 5, 1>, cms::soa::SoAColumnType::eigen, cms::soa::SoAAccessType::constAccess, 0, true>' requested here
              : public SoAColumnAccessorsImpl<T, columnType, accessType, alignment, restrictQualify> {
                       ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: note: in instantiation of template class 'cms::soa::SoAAccessors<Eigen::Matrix<float, 5, 1, 0, 5, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::constAccess>::Alignment<0UL>::RestrictQualifier<true>' requested here
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1543:25: note: expanded from macro 'GENERATE_SOA_LAYOUT'
        _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_CONST_ACCESSOR, ~, __VA_ARGS__)                                              \
                        ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:53:39: note: in instantiation of template class 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>' requested here
    auto start = (i == 0) ? 0 : tracks[i - 1].hitOffsets();
                                      ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: error: no type named 'ParamReturnType' in 'cms::soa::SoAAccessors<Eigen::Matrix<float, 5, 1, 0, 5, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::constAccess>::Alignment<0UL>::RestrictQualifier<true>'
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1543:25: note: expanded from macro 'GENERATE_SOA_LAYOUT'
        _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_CONST_ACCESSOR, ~, __VA_ARGS__)                                              \
        ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:671:65: note: expanded from macro '_ITERATE_ON_ALL'
#define _ITERATE_ON_ALL(MACRO, DATA, ...) BOOST_PP_SEQ_FOR_EACH(MACRO, DATA, BOOST_PP_VARIADIC_TO_SEQ(__VA_ARGS__))
                                          ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/seq/for_each.hpp:29:88: note: expanded from macro 'BOOST_PP_SEQ_FOR_EACH'
#    define BOOST_PP_SEQ_FOR_EACH(macro, data, seq) BOOST_PP_SEQ_FOR_EACH_DETAIL_CHECK(macro, data, seq)
                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
note: (skipping 51 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:18:60: note: expanded from macro 'BOOST_PP_IIF'
#    define BOOST_PP_IIF(bit, t, f) BOOST_PP_IIF_I(bit, t, f)
                                    ~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:25:63: note: expanded from macro 'BOOST_PP_IIF_I'
#    define BOOST_PP_IIF_I(bit, t, f) BOOST_PP_IIF_ ## bit(t, f)
                                      ~~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:31:31: note: expanded from macro 'BOOST_PP_IIF_0'
# define BOOST_PP_IIF_0(t, f) f
                              ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:53:39: note: in instantiation of template class 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>' requested here
    auto start = (i == 0) ? 0 : tracks[i - 1].hitOffsets();
                                      ^
In file included from DataFormatsEcalDigi_xr dictionary payload:76:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiHostCollection.h:5:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiSoA.h:6:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:11:
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:442:19: error: static assertion failed due to requirement '!sizeof(Eigen::Matrix<float, 15, 1, 0, 15, 1>)': Eigen/Core should be pre-included before the SoA headers to enable support for Eigen columns.
    static_assert(!sizeof(C),
                  ^~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:792:38: note: in instantiation of template class 'cms::soa::SoAValue<cms::soa::SoAColumnType::eigen, Eigen::Matrix<float, 15, 1, 0, 15, 1>, 0, true>' requested here
    using ParamReturnType = typename SoAValue<SoAColumnType::eigen, T, alignment, restrictQualify>::CMapType;
                                     ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:813:24: note: in instantiation of template class 'cms::soa::SoAColumnAccessorsImpl<Eigen::Matrix<float, 15, 1, 0, 15, 1>, cms::soa::SoAColumnType::eigen, cms::soa::SoAAccessType::constAccess, 0, true>' requested here
              : public SoAColumnAccessorsImpl<T, columnType, accessType, alignment, restrictQualify> {
                       ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: note: in instantiation of template class 'cms::soa::SoAAccessors<Eigen::Matrix<float, 15, 1, 0, 15, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::constAccess>::Alignment<0UL>::RestrictQualifier<true>' requested here
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1543:25: note: expanded from macro 'GENERATE_SOA_LAYOUT'
        _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_CONST_ACCESSOR, ~, __VA_ARGS__)                                              \
                        ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:53:39: note: in instantiation of template class 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>' requested here
    auto start = (i == 0) ? 0 : tracks[i - 1].hitOffsets();
                                      ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: error: no type named 'ParamReturnType' in 'cms::soa::SoAAccessors<Eigen::Matrix<float, 15, 1, 0, 15, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::constAccess>::Alignment<0UL>::RestrictQualifier<true>'
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1543:25: note: expanded from macro 'GENERATE_SOA_LAYOUT'
        _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_CONST_ACCESSOR, ~, __VA_ARGS__)                                              \
        ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:671:65: note: expanded from macro '_ITERATE_ON_ALL'
#define _ITERATE_ON_ALL(MACRO, DATA, ...) BOOST_PP_SEQ_FOR_EACH(MACRO, DATA, BOOST_PP_VARIADIC_TO_SEQ(__VA_ARGS__))
                                          ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/seq/for_each.hpp:29:88: note: expanded from macro 'BOOST_PP_SEQ_FOR_EACH'
#    define BOOST_PP_SEQ_FOR_EACH(macro, data, seq) BOOST_PP_SEQ_FOR_EACH_DETAIL_CHECK(macro, data, seq)
                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
note: (skipping 56 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:18:60: note: expanded from macro 'BOOST_PP_IIF'
#    define BOOST_PP_IIF(bit, t, f) BOOST_PP_IIF_I(bit, t, f)
                                    ~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:25:63: note: expanded from macro 'BOOST_PP_IIF_I'
#    define BOOST_PP_IIF_I(bit, t, f) BOOST_PP_IIF_ ## bit(t, f)
                                      ~~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:31:31: note: expanded from macro 'BOOST_PP_IIF_0'
# define BOOST_PP_IIF_0(t, f) f
                              ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:53:39: note: in instantiation of template class 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>' requested here
    auto start = (i == 0) ? 0 : tracks[i - 1].hitOffsets();
                                      ^
In file included from DataFormatsEcalDigi_xr dictionary payload:76:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiHostCollection.h:5:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiSoA.h:6:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:11:
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:561:19: error: static assertion failed due to requirement '!sizeof(Eigen::Matrix<float, 5, 1, 0, 5, 1>)': Eigen/Core should be pre-included before the SoA headers to enable support for Eigen columns.
    static_assert(!sizeof(C),
                  ^~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: note: in instantiation of template class 'cms::soa::SoAConstValue<cms::soa::SoAColumnType::eigen, Eigen::Matrix<float, 5, 1, 0, 5, 1>, 0, true>' requested here
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1522:25: note: expanded from macro 'GENERATE_SOA_LAYOUT'
        _ITERATE_ON_ALL(_DECLARE_VIEW_CONST_ELEMENT_ACCESSOR, ~, __VA_ARGS__)                                          \
                        ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:53:39: note: in instantiation of member class 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element' requested here
    auto start = (i == 0) ? 0 : tracks[i - 1].hitOffsets();
                                      ^
In file included from DataFormatsEcalDigi_xr dictionary payload:76:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiHostCollection.h:5:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiSoA.h:6:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:11:
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:561:19: error: static assertion failed due to requirement '!sizeof(Eigen::Matrix<float, 15, 1, 0, 15, 1>)': Eigen/Core should be pre-included before the SoA headers to enable support for Eigen columns.
    static_assert(!sizeof(C),
                  ^~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: note: in instantiation of template class 'cms::soa::SoAConstValue<cms::soa::SoAColumnType::eigen, Eigen::Matrix<float, 15, 1, 0, 15, 1>, 0, true>' requested here
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1522:25: note: expanded from macro 'GENERATE_SOA_LAYOUT'
        _ITERATE_ON_ALL(_DECLARE_VIEW_CONST_ELEMENT_ACCESSOR, ~, __VA_ARGS__)                                          \
                        ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:53:39: note: in instantiation of member class 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element' requested here
    auto start = (i == 0) ? 0 : tracks[i - 1].hitOffsets();
                                      ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:75:25: error: no member named 'state' in 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element'
    float v = tracks[i].state()(2);
              ~~~~~~~~~ ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:79:102: error: no member named 'state' in 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element'
  ALPAKA_FN_HOST_ACC inline float phi(const TrackSoAConstView &tracks, int32_t i) { return tracks[i].state()(0); }
                                                                                           ~~~~~~~~~ ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:81:102: error: no member named 'state' in 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element'
  ALPAKA_FN_HOST_ACC inline float tip(const TrackSoAConstView &tracks, int32_t i) { return tracks[i].state()(1); }
                                                                                           ~~~~~~~~~ ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:83:102: error: no member named 'state' in 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element'
  ALPAKA_FN_HOST_ACC inline float zip(const TrackSoAConstView &tracks, int32_t i) { return tracks[i].state()(4); }
                                                                                           ~~~~~~~~~ ^
In file included from DataFormatsEcalDigi_xr dictionary payload:76:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiHostCollection.h:5:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiSoA.h:6:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:11:
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:772:101: error: no type named 'MapType' in 'cms::soa::SoAValue<cms::soa::SoAColumnType::eigen, Eigen::Matrix<float, 5, 1, 0, 5, 1>, 0, true>'
    using ParamReturnType = typename SoAValue<SoAColumnType::eigen, T, alignment, restrictQualify>::MapType;
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:813:24: note: in instantiation of template class 'cms::soa::SoAColumnAccessorsImpl<Eigen::Matrix<float, 5, 1, 0, 5, 1>, cms::soa::SoAColumnType::eigen, cms::soa::SoAAccessType::mutableAccess, 0, true>' requested here
              : public SoAColumnAccessorsImpl<T, columnType, accessType, alignment, restrictQualify> {
                       ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: note: in instantiation of template class 'cms::soa::SoAAccessors<Eigen::Matrix<float, 5, 1, 0, 5, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::mutableAccess>::Alignment<0UL>::RestrictQualifier<true>' requested here
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1730:23: note: expanded from macro 'GENERATE_SOA_LAYOUT'
      _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_ACCESSOR, ~, __VA_ARGS__)                                                      \
                      ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:99:11: note: in instantiation of template class 'reco::TrackLayout<128, false>::ViewTemplateFreeParams<128, false, true, true>' requested here
    tracks[i].state() << cp.template cast<float>(), lp.template cast<float>();
          ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: error: no type named 'ParamReturnType' in 'cms::soa::SoAAccessors<Eigen::Matrix<float, 5, 1, 0, 5, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::mutableAccess>::Alignment<0UL>::RestrictQualifier<true>'
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1730:23: note: expanded from macro 'GENERATE_SOA_LAYOUT'
      _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_ACCESSOR, ~, __VA_ARGS__)                                                      \
      ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:671:65: note: expanded from macro '_ITERATE_ON_ALL'
#define _ITERATE_ON_ALL(MACRO, DATA, ...) BOOST_PP_SEQ_FOR_EACH(MACRO, DATA, BOOST_PP_VARIADIC_TO_SEQ(__VA_ARGS__))
                                          ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/seq/for_each.hpp:29:88: note: expanded from macro 'BOOST_PP_SEQ_FOR_EACH'
#    define BOOST_PP_SEQ_FOR_EACH(macro, data, seq) BOOST_PP_SEQ_FOR_EACH_DETAIL_CHECK(macro, data, seq)
                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
note: (skipping 51 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:18:60: note: expanded from macro 'BOOST_PP_IIF'
#    define BOOST_PP_IIF(bit, t, f) BOOST_PP_IIF_I(bit, t, f)
                                    ~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:25:63: note: expanded from macro 'BOOST_PP_IIF_I'
#    define BOOST_PP_IIF_I(bit, t, f) BOOST_PP_IIF_ ## bit(t, f)
                                      ~~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:31:31: note: expanded from macro 'BOOST_PP_IIF_0'
# define BOOST_PP_IIF_0(t, f) f
                              ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:99:11: note: in instantiation of template class 'reco::TrackLayout<128, false>::ViewTemplateFreeParams<128, false, true, true>' requested here
    tracks[i].state() << cp.template cast<float>(), lp.template cast<float>();
          ^
In file included from DataFormatsEcalDigi_xr dictionary payload:76:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiHostCollection.h:5:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/EcalDigi/interface/EcalDigiSoA.h:6:
In file included from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:11:
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:772:101: error: no type named 'MapType' in 'cms::soa::SoAValue<cms::soa::SoAColumnType::eigen, Eigen::Matrix<float, 15, 1, 0, 15, 1>, 0, true>'
    using ParamReturnType = typename SoAValue<SoAColumnType::eigen, T, alignment, restrictQualify>::MapType;
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:813:24: note: in instantiation of template class 'cms::soa::SoAColumnAccessorsImpl<Eigen::Matrix<float, 15, 1, 0, 15, 1>, cms::soa::SoAColumnType::eigen, cms::soa::SoAAccessType::mutableAccess, 0, true>' requested here
              : public SoAColumnAccessorsImpl<T, columnType, accessType, alignment, restrictQualify> {
                       ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: note: in instantiation of template class 'cms::soa::SoAAccessors<Eigen::Matrix<float, 15, 1, 0, 15, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::mutableAccess>::Alignment<0UL>::RestrictQualifier<true>' requested here
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1730:23: note: expanded from macro 'GENERATE_SOA_LAYOUT'
      _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_ACCESSOR, ~, __VA_ARGS__)                                                      \
                      ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:99:11: note: in instantiation of template class 'reco::TrackLayout<128, false>::ViewTemplateFreeParams<128, false, true, true>' requested here
    tracks[i].state() << cp.template cast<float>(), lp.template cast<float>();
          ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:21:3: error: no type named 'ParamReturnType' in 'cms::soa::SoAAccessors<Eigen::Matrix<float, 15, 1, 0, 15, 1> >::ColumnType<cms::soa::SoAColumnType::eigen>::AccessType<cms::soa::SoAAccessType::mutableAccess>::Alignment<0UL>::RestrictQualifier<true>'
  GENERATE_SOA_LAYOUT(TrackLayout,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoALayout.h:1730:23: note: expanded from macro 'GENERATE_SOA_LAYOUT'
      _ITERATE_ON_ALL(_DECLARE_VIEW_SOA_ACCESSOR, ~, __VA_ARGS__)                                                      \
      ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/SoATemplate/interface/SoACommon.h:671:65: note: expanded from macro '_ITERATE_ON_ALL'
#define _ITERATE_ON_ALL(MACRO, DATA, ...) BOOST_PP_SEQ_FOR_EACH(MACRO, DATA, BOOST_PP_VARIADIC_TO_SEQ(__VA_ARGS__))
                                          ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/seq/for_each.hpp:29:88: note: expanded from macro 'BOOST_PP_SEQ_FOR_EACH'
#    define BOOST_PP_SEQ_FOR_EACH(macro, data, seq) BOOST_PP_SEQ_FOR_EACH_DETAIL_CHECK(macro, data, seq)
                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
note: (skipping 56 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:18:60: note: expanded from macro 'BOOST_PP_IIF'
#    define BOOST_PP_IIF(bit, t, f) BOOST_PP_IIF_I(bit, t, f)
                                    ~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:25:63: note: expanded from macro 'BOOST_PP_IIF_I'
#    define BOOST_PP_IIF_I(bit, t, f) BOOST_PP_IIF_ ## bit(t, f)
                                      ~~~~~~~~~~~~~~~~~~~~~~~~^~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include/boost/preprocessor/control/iif.hpp:31:31: note: expanded from macro 'BOOST_PP_IIF_0'
# define BOOST_PP_IIF_0(t, f) f
                              ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:99:11: note: in instantiation of template class 'reco::TrackLayout<128, false>::ViewTemplateFreeParams<128, false, true, true>' requested here
    tracks[i].state() << cp.template cast<float>(), lp.template cast<float>();
          ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:127:19: error: no member named 'state' in 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element'
    v = tracks[i].state().template cast<typename V5::Scalar>();
        ~~~~~~~~~ ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:129:29: error: no member named 'covariance' in 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element'
      cov(j, j) = tracks[i].covariance()(ind++);
                  ~~~~~~~~~ ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/src/DataFormats/TrackSoA/interface/TracksSoA.h:131:43: error: no member named 'covariance' in 'reco::TrackLayout<128, false>::ConstViewTemplateFreeParams<128, false, true, true>::const_element'
        cov(k, j) = cov(j, k) = tracks[i].covariance()(ind++);
                                ~~~~~~~~~ ^

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x000014f815b8eeeb in waitpid () from /lib64/libc.so.6
#1  0x000014f815ae2747 in do_system () from /lib64/libc.so.6
#2  0x000014f816b87965 in TUnixSystem::StackTrace() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#3  0x000014f816b87314 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#4  <signal handler called>
#5  0x000014f8121c960b in clang::BaseUsingDecl::removeShadowDecl(clang::UsingShadowDecl*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#6  0x000014f80edf7f5b in clang::declvisitor::Base<std::add_pointer, cling::DeclUnloader, bool>::Visit(clang::Decl*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#7  0x000014f80edf4545 in cling::DeclUnloader::VisitDeclContext(clang::DeclContext*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#8  0x000014f80edf469c in cling::DeclUnloader::VisitFunctionDecl(clang::FunctionDecl*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#9  0x000014f80edb6815 in cling::TransactionUnloader::unloadDeserializedDeclarations(cling::Transaction*, cling::DeclUnloader&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#10 0x000014f80edb6b0f in cling::TransactionUnloader::RevertTransaction(cling::Transaction*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#11 0x000014f80ed94ff3 in cling::Interpreter::unload(cling::Transaction&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#12 0x000014f80ee2e2ed in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#13 0x000014f80ee3120b in cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#14 0x000014f80ed96742 in cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#15 0x000014f80ec9eaac in ExecAutoParse(char const*, bool, cling::Interpreter*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#16 0x000014f80eca133b in TCling::AutoParseImplRecurse(char const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#17 0x000014f80ecaa619 in TCling::AutoParse(char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#18 0x000014f816b0d9dd in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#19 0x000014f7f6495978 in TTreeFormula::BranchHasMethod(TLeaf*, TBranch*, char const*, char const*, long long) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#20 0x000014f7f64972d0 in TTreeFormula::FindLeafForExpression(char const*, TLeaf*&, TString&, bool&, unsigned int&, TObjArray&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, bool&, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#21 0x000014f7f6499212 in TTreeFormula::DefinedVariable(TString&, int&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#22 0x000014f7f966e5ef in ROOT::v5::TFormula::Analyze(char const*, int&, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libHist.so
#23 0x000014f7f96776dd in ROOT::v5::TFormula::Compile(char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libHist.so
#24 0x000014f7f6493bfd in TTreeFormula::Init(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#25 0x000014f7f6494399 in TTreeFormula::TTreeFormula(char const*, char const*, TTree*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#26 0x000014f7f647ac14 in TSelectorDraw::CompileVariables(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#27 0x000014f7f6475ce4 in TSelectorDraw::Begin(TTree*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#28 0x000014f7f64ae04b in TTreePlayer::Process(TSelector*, char const*, long long, long long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#29 0x000014f7f64aef6c in TTreePlayer::DrawSelect(char const*, char const*, char const*, long long, long long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#30 0x000014f7fb06351a in plotvar(TString, TString, bool) () from /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/validate_lib/validate_C.so
#31 0x000014f7fb07e62f in validateEvents(TString, TString, TString, TString, bool, TString) () from /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/validate_lib/validate_C.so
#32 0x000014f7fb0aaf61 in validate(TString, TString, TString, TString, bool, TString) () from /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/validate_lib/validate_C.so
#33 0x000014f8155e00d7 in ?? ()
#34 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#35 0x000000006c6c6106 in ?? ()
#36 0x0000003201000000 in ?? ()
#37 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#38 0x0000007100000081 in ?? ()
#39 0x0000000003f88340 in ?? ()
#40 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#41 0x0000008400000091 in ?? ()
#42 0x000000000577cea0 in ?? ()
#43 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#44 0x00000000544c4806 in ?? ()
#45 0x000014f800000040 in ?? ()
#46 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#47 0x0000000000000000 in ?? ()
===========================================================


The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
preferably using the command (.forum bug) in the ROOT prompt.
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs or (preferably) using the command (.gh bug) in
the ROOT prompt. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x000014f8121c960b in clang::BaseUsingDecl::removeShadowDecl(clang::UsingShadowDecl*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#6  0x000014f80edf7f5b in clang::declvisitor::Base<std::add_pointer, cling::DeclUnloader, bool>::Visit(clang::Decl*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#7  0x000014f80edf4545 in cling::DeclUnloader::VisitDeclContext(clang::DeclContext*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#8  0x000014f80edf469c in cling::DeclUnloader::VisitFunctionDecl(clang::FunctionDecl*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#9  0x000014f80edb6815 in cling::TransactionUnloader::unloadDeserializedDeclarations(cling::Transaction*, cling::DeclUnloader&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#10 0x000014f80edb6b0f in cling::TransactionUnloader::RevertTransaction(cling::Transaction*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#11 0x000014f80ed94ff3 in cling::Interpreter::unload(cling::Transaction&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#12 0x000014f80ee2e2ed in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#13 0x000014f80ee3120b in cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#14 0x000014f80ed96742 in cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#15 0x000014f80ec9eaac in ExecAutoParse(char const*, bool, cling::Interpreter*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#16 0x000014f80eca133b in TCling::AutoParseImplRecurse(char const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#17 0x000014f80ecaa619 in TCling::AutoParse(char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-07-2300/external/el8_amd64_gcc13/lib/libCling.so
#18 0x000014f816b0d9dd in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#19 0x000014f7f6495978 in TTreeFormula::BranchHasMethod(TLeaf*, TBranch*, char const*, char const*, long long) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#20 0x000014f7f64972d0 in TTreeFormula::FindLeafForExpression(char const*, TLeaf*&, TString&, bool&, unsigned int&, TObjArray&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, bool&, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#21 0x000014f7f6499212 in TTreeFormula::DefinedVariable(TString&, int&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#22 0x000014f7f966e5ef in ROOT::v5::TFormula::Analyze(char const*, int&, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libHist.so
#23 0x000014f7f96776dd in ROOT::v5::TFormula::Compile(char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libHist.so
#24 0x000014f7f6493bfd in TTreeFormula::Init(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#25 0x000014f7f6494399 in TTreeFormula::TTreeFormula(char const*, char const*, TTree*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#26 0x000014f7f647ac14 in TSelectorDraw::CompileVariables(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#27 0x000014f7f6475ce4 in TSelectorDraw::Begin(TTree*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#28 0x000014f7f64ae04b in TTreePlayer::Process(TSelector*, char const*, long long, long long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#29 0x000014f7f64aef6c in TTreePlayer::DrawSelect(char const*, char const*, char const*, long long, long long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/lib/libTreePlayer.so
#30 0x000014f7fb06351a in plotvar(TString, TString, bool) () from /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/validate_lib/validate_C.so
#31 0x000014f7fb07e62f in validateEvents(TString, TString, TString, TString, bool, TString) () from /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/validate_lib/validate_C.so
#32 0x000014f7fb0aaf61 in validate(TString, TString, TString, TString, bool, TString) () from /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/validate_lib/validate_C.so
#33 0x000014f8155e00d7 in ?? ()
#34 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#35 0x000000006c6c6106 in ?? ()
#36 0x0000003201000000 in ?? ()
#37 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#38 0x0000007100000081 in ?? ()
#39 0x0000000003f88340 in ?? ()
#40 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#41 0x0000008400000091 in ?? ()
#42 0x000000000577cea0 in ?? ()
#43 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#44 0x00000000544c4806 in ?? ()
#45 0x000014f800000040 in ?? ()
#46 0x000014f816c690d8 in vtable for TString () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/bin/../lib/libCore.so
#47 0x0000000000000000 in ?? ()
===========================================================

mmusich avatar Dec 08 '25 14:12 mmusich

cms-bot internal usage

cmsbuild avatar Dec 08 '25 14:12 cmsbuild

A new Issue was created by @mmusich.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

cmsbuild avatar Dec 08 '25 14:12 cmsbuild

I wonder if this is related to having SoA in the HLT output (e.g. issue https://github.com/cms-sw/cmssw/issues/49333 and https://github.com/cms-sw/cmssw/issues/49349).

mmusich avatar Dec 08 '25 14:12 mmusich

assign core, heterogeneous

makortel avatar Dec 08 '25 15:12 makortel

type root

makortel avatar Dec 08 '25 15:12 makortel

New categories assigned: core,heterogeneous

@Dr15Jones,@fwyzard,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Dec 08 '25 15:12 cmsbuild

The symptoms smell the as in the other recent issues with missing PortableHostCollection-related dictionaries. From the log the culprit seems to be EcalDigiHostCollection, and indeed the EcalDigiHostCollection::Layout is not specified in the classes_def.xml https://github.com/cms-sw/cmssw/blob/311b2a402ba3ec0907adb765b4407085dd669bda/DataFormats/EcalDigi/src/classes_def.xml#L151-L156

makortel avatar Dec 08 '25 15:12 makortel

The symptoms smell the as in the other recent issues with missing PortableHostCollection-related dictionaries. From the log the culprit seems to be EcalDigiHostCollection, and indeed the EcalDigiHostCollection::Layout is not specified in the classes_def.xml

Would:

diff --git a/DataFormats/EcalDigi/src/classes_def.xml b/DataFormats/EcalDigi/src/classes_def.xml
index 22036e9446e..3d04c6e2064 100644
--- a/DataFormats/EcalDigi/src/classes_def.xml
+++ b/DataFormats/EcalDigi/src/classes_def.xml
@@ -150,6 +150,7 @@
    <class name="EcalDataArray"/>
    <class name="EcalDigiSoA"/>
    <class name="EcalDigiSoA::View"/>
+   <class name="EcalDigiHostCollection::Layout"/>
    <class name="EcalDigiHostCollection" ClassVersion="3">
      <version ClassVersion="3" checksum="9207964"/>
    </class>

be enough?

mmusich avatar Dec 09 '25 11:12 mmusich

Almost. The

<class name="EcalDigiHostCollection::Layout"/>

must (presently) be before the type it aliases, i.e.

<class name="EcalDigiSoA"/>

(so I'd suggest add a comment about that)

On the same go, I'd suggest to do the same for EcalDigiPhase2HostCollection.

A further, although independent, cleanup would be to remove the class versions from EcalDigiHostCollection and EcalDigiPhase2HostCollection (PortableHostCollection is a template class, for the class versioning for those would need to be done differently).

makortel avatar Dec 09 '25 15:12 makortel

Thanks! I opened https://github.com/cms-sw/cmssw/pull/49581, let's follow up there.

mmusich avatar Dec 09 '25 15:12 mmusich

As noted in https://github.com/cms-sw/cmssw/pull/49581#issuecomment-3635924842, adding the EcalDigiHostCollection::Layout was not effective. I started a ROOT debug build in https://github.com/cms-sw/cmsdist/pull/10212

makortel avatar Dec 10 '25 14:12 makortel

After figuring out what exactly validateJR does with ROOT https://github.com/cms-sw/cms-bot/blob/2843bde01d57f577e0570d0e0c46e066f1059a6e/comparisons/validateJR.py#L66-L70 and making a test program to replicate that (ROOT prompt apparently exits in a way that does not trigger a break in an outer gdb) I got an assertion failure

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/llvm-project/clang/lib/AST/DeclCXX.cpp:3136: void cl
ang::BaseUsingDecl::removeShadowDecl(clang::UsingShadowDecl*): Assertion `llvm::is_contained(shadows(), S) && "declaration not in set"' failed.

#0  0x00007ffff394c52f in raise () from /lib64/libc.so.6                                                                                                                                                      [0/1513]
#1  0x00007ffff391fe65 in abort () from /lib64/libc.so.6
#2  0x00007ffff391fd39 in __assert_fail_base.cold.0 () from /lib64/libc.so.6
#3  0x00007ffff3944e86 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fffe9ffd0b4 in clang::BaseUsingDecl::removeShadowDecl (this=0x2f5c0210, S=0x2f5c03d8)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/llvm-project/clang/lib/AST/DeclCXX.cpp:3136
#5  0x00007fffe4b7aba0 in cling::DeclUnloader::VisitUsingShadowDecl (this=0x7ffffffca830, USD=0x2f5c03d8)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/DeclUnloader.cpp:588
#6  0x00007fffe4af1ab4 in clang::declvisitor::Base<std::add_pointer, cling::DeclUnloader, bool>::Visit (this=0x7ffffffca830, D=0x2f5c03d8)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/build/interpreter/llvm-project/llvm/tools/clang/include/clang/AST/DeclNodes.inc:370
#7  0x00007fffe4b7b34e in cling::DeclUnloader::VisitDeclContext (this=0x7ffffffca830, DC=0x2f5c0028)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/DeclUnloader.cpp:777
#8  0x00007fffe4b7ae9e in cling::DeclUnloader::VisitFunctionDecl (this=0x7ffffffca830, FD=0x2f5bffe0, RemoveSpec=true)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/DeclUnloader.cpp:647
#9  0x00007fffe4b7b034 in cling::DeclUnloader::VisitFunctionDecl (this=0x7ffffffca830, FD=0x2f5bffe0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/DeclUnloader.cpp:710
#10 0x00007fffe4af18ec in clang::declvisitor::Base<std::add_pointer, cling::DeclUnloader, bool>::Visit (this=0x7ffffffca830, D=0x2f5bffe0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/build/interpreter/llvm-project/llvm/tools/clang/include/clang/AST/DeclNodes.inc:238
#11 0x00007fffe4aef394 in cling::DeclUnloader::UnloadDecl (this=0x7ffffffca830, D=0x2f5bffe0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/DeclUnloader.h:65
#12 0x00007fffe4afbe99 in cling::TransactionUnloader::unloadDeserializedDeclarations (this=0x7ffffffca8f0, T=0x2b6874b0, DeclU=...)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/TransactionUnloader.cpp:100
#13 0x00007fffe4afc223 in cling::TransactionUnloader::RevertTransaction (this=0x7ffffffca8f0, T=0x2b6874b0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/TransactionUnloader.cpp:148
#14 0x00007fffe4acefba in cling::Interpreter::unload (this=0x4770b0, T=...)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/Interpreter.cpp:1573
#15 0x00007fffe4befc9f in cling::IncrementalParser::commitTransaction (this=0x587db0, PRT=..., ClearDiagClient=true)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/IncrementalParser.cpp:610
#16 0x00007fffe4bf08ab in cling::IncrementalParser::Compile (this=0x587db0, input=..., Opts=...)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/IncrementalParser.cpp:850
#17 0x00007fffe4ace064 in cling::Interpreter::DeclareInternal (this=0x4770b0, input=..., CO=..., T=0x0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/Interpreter.cpp:1375
#18 0x00007fffe4acc9af in cling::Interpreter::parseForModule (this=0x4770b0, input=...)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/interpreter/cling/lib/Interpreter/Interpreter.cpp:965
#19 0x00007fffe480e730 in ExecAutoParse (
    what=0x7fffce95a000 "\n#line 1 \"DataFormatsTrackSoA_xr dictionary payload\"\n\n#ifndef ALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT\n  #define ALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT 128\n#endif\n#ifndef ALPAKA_DISABLE_VENDOR_RNG\n  #define"..., header=false, interpreter=0x4770b0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/core/metacling/src/TCling.cxx:6403
#20 0x00007fffe480efb3 in TCling::AutoParseImplRecurse (this=0x586730, cls=0x8c4d5c0 "edm::Wrapper<PortableHostMultiCollection<reco::TrackLayout<128,false>,reco::TrackHitsLayout<128,false> > >", topLevel=true)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/core/metacling/src/TCling.cxx:6508
#21 0x00007fffe480f625 in TCling::AutoParse (this=0x586730, cls=0x8c4d5c0 "edm::Wrapper<PortableHostMultiCollection<reco::TrackLayout<128,false>,reco::TrackHitsLayout<128,false> > >")
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/core/metacling/src/TCling.cxx:6593
#22 0x00007ffff7bd00f2 in TClass::LoadClassInfo (this=0x8c44cb0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/core/meta/src/TClass.cxx:5976
#23 0x00007ffff7af6874 in TClass::GetClassInfo (this=0x8c44cb0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/core/meta/inc/TClass.h:442
#24 0x00007ffff61257c9 in TTreeFormula::BranchHasMethod (this=0x16fcf9c0, leafcur=0x13d52f90, branch=0x13d51140, method=0x7ffffffd1200 "edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.size",
    params=0x7ffffffd1237 ")", readentry=0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TTreeFormula.cxx:3348
#25 0x00007ffff6120847 in TTreeFormula::FindLeafForExpression (this=0x16fcf9c0, expression=0x7ffffffd4e00 "[email protected]()", leaf=@0x7ffffffd4588: 0x0, leftover=...,
    final=@0x7ffffffd45ff: false, paran_level=@0x7ffffffd45f8: 0, castqueue=..., aliasUsed=..., useLeafCollectionObject=@0x7ffffffd45af: false,
    fullExpression=0x16faa070 "[email protected]()")
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TTreeFormula.cxx:2302
#26 0x00007ffff6123220 in TTreeFormula::DefinedVariable (this=0x16fcf9c0, name=..., action=@0x7ffffffd5814: 150)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TTreeFormula.cxx:2866
#27 0x00007ffff6debf4a in ROOT::v5::TFormula::Analyze (this=0x16fcf9c0, schain=0x16fab890 "[email protected]()", err=@0x7ffffffd6ca8: 0, offset=0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/hist/hist/src/TFormula_v5.cxx:1382
#28 0x00007ffff6df4574 in ROOT::v5::TFormula::Compile (this=0x16fcf9c0, expression=0x16fab720 "[email protected]()")
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/hist/hist/src/TFormula_v5.cxx:2369
#29 0x00007ffff6116776 in TTreeFormula::Init (this=0x16fcf9c0, name=0x16f77730 "Var1", expression=0x16fab720 "[email protected]()")
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TTreeFormula.cxx:221
#30 0x00007ffff611612f in TTreeFormula::TTreeFormula (this=0x16fcf9c0, name=0x16f77730 "Var1", expression=0x16fab720 "[email protected]()", tree=0x1ee3300)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TTreeFormula.cxx:174
#31 0x00007ffff6105abb in TSelectorDraw::CompileVariables (this=0x16fb8ca0, varexp=0x16f86520 "[email protected]()", selection=0x7ffffffd74b1 "")
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TSelectorDraw.cxx:1028
#32 0x00007ffff61004c0 in TSelectorDraw::Begin (this=0x16fb8ca0, tree=0x1ee3300)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TSelectorDraw.cxx:483
#33 0x00007ffff6154e71 in TTreePlayer::Process (this=0x16f8bf60, selector=0x16fb8ca0, option=0x7ffff457b221 "", nentries=10, firstentry=0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TTreePlayer.cxx:2251
#34 0x00007ffff614ab89 in TTreePlayer::DrawSelect (this=0x16f8bf60,
    varexp0=0x16f56460 "[email protected]()>>edmErrorSummaryEntrys_logErrorHarvester__RECO_objAT_size_refplot", selection=0x7ffffffd7aa1 "", option=0x7ffff457b221 "",
    nentries=10, firstentry=0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/treeplayer/src/TTreePlayer.cxx:428
#35 0x00007ffff6441882 in TTree::Draw (this=0x1ee3300, varexp=0x16f56460 "[email protected]()>>edmErrorSummaryEntrys_logErrorHarvester__RECO_objAT_size_refplot",
    selection=0x7ffffffd7aa1 "", option=0x7ffff457b221 "", nentries=10, firstentry=0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/tree/src/TTree.cxx:4519
#36 0x00007ffff6441802 in TTree::Draw (this=0x1ee3300, varexp=0x16f56460 "[email protected]()>>edmErrorSummaryEntrys_logErrorHarvester__RECO_objAT_size_refplot",
    selection=..., option=0x7ffff457b221 "", nentries=10, firstentry=0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc13/lcg/root/6.36.07-2b30384a88e536334f54f9c6dec34451/root-6.36.07/tree/tree/src/TTree.cxx:3865
#37 0x00007ffff4518433 in plotvar (v=..., cut=..., tryCatch=false) at /build/mkortela/debug/issue49568/CMSSW_16_0_X_2025-12-10-1100/src/validate_lib/./validate.C:114
#38 0x00007ffff4539b3a in validateEvents (step=..., file=..., refFile=..., r=..., SHOW=false, sr=...) at /build/mkortela/debug/issue49568/CMSSW_16_0_X_2025-12-10-1100/src/validate_lib/./validate.C:1070
#39 0x00007ffff457941e in validate (step=..., file=..., refFile=..., r=..., SHOW=false, sr=...) at /build/mkortela/debug/issue49568/CMSSW_16_0_X_2025-12-10-1100/src/validate_lib/./validate.C:2726
#40 0x000000000040124c in main ()

So the program is processing [email protected]() expression, and somehow that leads to TCling::AutoParse() call for edm::Wrapper<PortableHostMultiCollection<reco::TrackLayout<128,false>,reco::TrackHitsLayout<128,false> > > despite of https://github.com/cms-sw/cmssw/pull/49360.

makortel avatar Dec 10 '25 23:12 makortel

The TClass in stack frame #23 0x00007ffff7af6874 in TClass::GetClassInfo (this=0x8c44cb0) is for edm::Wrapper<PortableHostMultiCollection<reco::TrackLayout<128,false>,reco::TrackHitsLayout<128,false> > >

makortel avatar Dec 10 '25 23:12 makortel

In stack frame #24 0x00007ffff61257c9 in TTreeFormula::BranchHasMethod the branch argument is for 128falserecoTrackLayout128falserecoTrackHitsLayoutPortableHostMultiCollection_hltPixelTracksSoA__HLT.

In stack frame #25 0x00007ffff6120847 in TTreeFormula::FindLeafForExpression the code https://github.com/root-project/root/blob/3a3891e8b30512fca238abf66b74ce4459f3b37c/tree/treeplayer/src/TTreeFormula.cxx#L2296-L2307 looks like it iterates all leaves of the TTree to check if they have the method in question.

makortel avatar Dec 11 '25 18:12 makortel

I don't have good ideas how to proceed.

@pcanal @vepadulano Do you have any suggestions if TTreeFormula can be used when one of the TTree branches, that is not used in the TTreeFormula expression, has a type whose header parsing fails?

One option to make the SoALayout.h parseable would be to make Eigen a hard dependence for the SoAs. That option was also discussed in https://github.com/cms-sw/cmssw/issues/49333#issuecomment-3517586999.

Theoretically it would be nice if the SoA system would be extensible such that additional column types could be added in additional headers, but I have no idea how (in)feasible that would be.

makortel avatar Dec 11 '25 22:12 makortel

+core

Because the problem is not in the IO.

makortel avatar Dec 11 '25 22:12 makortel

assign reconstruction

I'm not sure if reco really maintains validateJR (if not, who does?), but the results are reported as "reco comparison differences" so presumably they should be at least aware.

makortel avatar Dec 11 '25 22:12 makortel

New categories assigned: reconstruction

@jfernan2,@mandrenguyen,@Moanwar,@srimanob you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Dec 11 '25 22:12 cmsbuild

No, from RECO we don't mantain validateJR, could it belong to PPD?

jfernan2 avatar Dec 12 '25 10:12 jfernan2

No, from RECO we don't mantain validateJR,

arguably it should -- though.

mmusich avatar Dec 12 '25 10:12 mmusich

I don't have good ideas how to proceed.

perhaps we can just skip all SoA-s at this point in time. Tagging also @slava77 and @vlimant as from the commit history they appear to be de facto maintainers of the tool.

mmusich avatar Dec 12 '25 10:12 mmusich

I don't have good ideas how to proceed.

perhaps we can just skip all SoA-s at this point in time. Tagging also @slava77 and @vlimant as from the commit history they appear to be de facto maintainers of the tool.

this was under RECO maintenance since 2010 or was it even earlier. I guess if it's not in the RECO mandate, RECO L2s are free to abandon. There can be other ways to monitor RECO outputs.

slava77 avatar Dec 12 '25 13:12 slava77

guess if it's not in the RECO mandate, RECO L2s are free to abandon.

If confirmed, would this mean we could stop running validateJR in PR tests?

makortel avatar Dec 12 '25 14:12 makortel

perhaps we can just skip all SoA-s at this point in time.

If I understood the ROOT code correctly, already the existence of the SoAs in the TTree would be enough to cause any(?) TTreeFormula on that TTree to fail like this. Just to make sure, are you suggesting

  • to stop storing SoAs in the files that are passed to validateJR,
  • clone the files passed to validateJR somehow filtering out the SoAs,
  • stop running validateJR on the files that have SoAs, or
  • something else

?

makortel avatar Dec 12 '25 14:12 makortel

either

  • clone the files passed to validateJR somehow filtering out the SoAs,

or

  • stop running validateJR on the files that have SoAs

with a preference for the former, to keep monitoring the rest of the products in the event.

mmusich avatar Dec 12 '25 14:12 mmusich

We (@Moanwar and I) will try to have a look, we were not aware of this tool since, AFAIU, it was used for RelMon mainly. RECO does not use it directly

jfernan2 avatar Dec 12 '25 14:12 jfernan2

The tool is mentioned in https://twiki.cern.ch/twiki/bin/viewauth/CMS/RecoIntegration#Compare_outputs_FWLite_style

makortel avatar Dec 12 '25 15:12 makortel

The tool is mentioned in https://twiki.cern.ch/twiki/bin/viewauth/CMS/RecoIntegration#Compare_outputs_FWLite_style

Yes, it was used by previous RECO conveners as Slava mentioned previously. We don't use it at present beyond the PR comparison. We will have a look anyway, @Moanwar is looking into it indeed.

jfernan2 avatar Dec 12 '25 16:12 jfernan2

No, from RECO we don't mantain validateJR, could it belong to PPD?

I touched the file a couple of month back maybe, after writing it a decade ago :-) but it's definitely not PPD. the script is a simple branch to branch comparison script using fwlite. I, and probably find it quite useful in PR testing to check the direct effect of software changes on sim/reco/aod/mini/nano products ; if that validateJR has to go away, CMS would certainly need to find something that does about the same content/content comparison to visual potential changes.

vlimant avatar Dec 12 '25 17:12 vlimant

one way to transfer the responsibility to PPD is to ensure that all products are monitored in some DQM module (or at least the dependent products are monitored). This way the FWLite validate.C is not as essential.

It does carry an additional ability to test the schema evolution by virtue of having the old and new files loaded at the same time. This part is rather sparsely covered with the DQM tools.

slava77 avatar Dec 12 '25 20:12 slava77