cms-sw
Moved storage related code to FWStorage subsystem
PR description:
- Reorganized storage related code to be in a new dedicated FWStorage subdirectory. This contains code which is TTree/RNTuple agnostic.
- Moved storage related code out of FWCore/Services to FWStorage/Services to better isolate external dependencies.
This work is based on the discussion here https://github.com/cms-sw/framework-team/issues/1256#issuecomment-3617363107
PR validation:
Framework tests mostly work. The ones that fail are because of an incompatibility between CMS partial builds and how ROOT handles its rootmap files. When ROOT parses the rootmap files, it lumps all lib*.so files from different directories into the same list and complains if there are overlaps. Therefore if we move a dictionary from one package to a different one, ROOT will see both placements, one in the user work area and the other coming from the CMSSW release area. Scram avoids this problem by taking into account LD_LIBRARY_PATH ordering but ROOT does not appear to do that.
resolves https://github.com/cms-sw/framework-team/issues/1715
-code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47132
Code check has found code style and quality issues which could be resolved by applying following patch(s)
- code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47132/code-format.patch
e.g.
curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47132/code-format.patch | patch -p1You can also runscram build code-formatto apply code format directly
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47133
A new Pull Request was created by @Dr15Jones for master.
It involves the following packages:
- FWCore/ParameterSet (core)
- FWCore/Services (core)
- FWStorage/DCacheAdaptor (****)
- FWStorage/Services (****)
- FWStorage/StorageFactory (****)
- FWStorage/TFileAdaptor (****)
- FWStorage/XrdAdaptor (****)
- IOPool/Common (core)
- IOPool/Input (core)
- IOPool/Streamer (core)
- IOPool/TFileAdaptor (core)
- Utilities/DCacheAdaptor (core)
- Utilities/StorageFactory (core)
- Utilities/XrdAdaptor (core)
The following packages do not have a category, yet:
FWStorage/DCacheAdaptor FWStorage/Services FWStorage/StorageFactory FWStorage/TFileAdaptor FWStorage/XrdAdaptor Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category
@Dr15Jones, @makortel, @smuzaffar can you please review it and eventually sign? Thanks. @fwyzard, @makortel, @wddgit this is something you requested to watch as well. @ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47134
Pull request #49594 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please check and sign again.
-1
Failed Tests: Build HeaderConsistency
Size: This PR adds an extra 412KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49888/summary.html
COMMIT: c087667891c9846907f7eb89021408cf3aac0b18
CMSSW: CMSSW_16_0_X_2025-12-10-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49594/49888/install.sh to create a dev area with all the needed externals and cmssw changes.
The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
- @stahlleiton cms-sw/cmssw#49561
You can see more details here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49888/git-recent-commits.json https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49888/git-merge-result
Failed Build
I found compilation error when building:
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/bin/c++ -c -DCMS_MICRO_ARCH='x86-64-v3' -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=130400 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -DBOOST_MPL_IGNORE_PARENTHESES_WARNING -DCMSSW_GIT_HASH='CMSSW_16_0_X_2025-12-10-1100' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_16_0_X_2025-12-10-1100' -Isrc -Ipoison -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-10-1100/src -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/pcre/8.43-6d98fda3bfd074ebb583e2d6a2c75d25/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/bz2lib/1.0.6-d113e1c6278c07eeaff5f84db9548446/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/libuuid/2.34-5ba7a8abfc0c5fecdc448cca360c25ff/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/tbb/v2022.3.0-54089f7aad511dfcd272d65c5ee4b3ab/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/xz/5.6.4-b9c4ffbc390ed320a5d57fd552e29a05/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/zlib/1.2.13-589f6bb51bbeba38a7adf5a10ea8a093/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/md5/1.0.0-26057075013e190e56dad37d35219376/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/tinyxml2/6.2.0-67924ead96ecb4e69aad321b767979a5/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el8_amd64_gcc13/src/IORawData/SiPixelInputSources/src/IORawDataSiPixelInputSources/PixelSLinkDataInputSource.cc.d src/IORawData/SiPixelInputSources/src/PixelSLinkDataInputSource.cc -o tmp/el8_amd64_gcc13/src/IORawData/SiPixelInputSources/src/IORawDataSiPixelInputSources/PixelSLinkDataInputSource.cc.o
>> Compiling edm plugin src/IORawData/SiPixelInputSources/src/SealModule.cc
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/bin/c++ -c -DCMS_MICRO_ARCH='x86-64-v3' -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=130400 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -DBOOST_MPL_IGNORE_PARENTHESES_WARNING -DCMSSW_GIT_HASH='CMSSW_16_0_X_2025-12-10-1100' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_16_0_X_2025-12-10-1100' -Isrc -Ipoison -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-12-10-1100/src -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/pcre/8.43-6d98fda3bfd074ebb583e2d6a2c75d25/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/boost/1.80.0-6fb83b2bac2398768f2cc6374a639c37/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/bz2lib/1.0.6-d113e1c6278c07eeaff5f84db9548446/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/libuuid/2.34-5ba7a8abfc0c5fecdc448cca360c25ff/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/lcg/root/6.36.07-0490e78b33943d48f9af1c19441a9d2e/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/tbb/v2022.3.0-54089f7aad511dfcd272d65c5ee4b3ab/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/xz/5.6.4-b9c4ffbc390ed320a5d57fd552e29a05/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/zlib/1.2.13-589f6bb51bbeba38a7adf5a10ea8a093/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/md5/1.0.0-26057075013e190e56dad37d35219376/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el8_amd64_gcc13/external/tinyxml2/6.2.0-67924ead96ecb4e69aad321b767979a5/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el8_amd64_gcc13/src/IORawData/SiPixelInputSources/src/IORawDataSiPixelInputSources/SealModule.cc.d src/IORawData/SiPixelInputSources/src/SealModule.cc -o tmp/el8_amd64_gcc13/src/IORawData/SiPixelInputSources/src/IORawDataSiPixelInputSources/SealModule.cc.o
In file included from src/IORawData/SiPixelInputSources/interface/PixelSLinkDataInputSource.h:33,
from src/IORawData/SiPixelInputSources/src/PixelSLinkDataInputSource.cc:24:
poison/Utilities/StorageFactory/interface/Storage.h:1:2: error: #error THIS FILE HAS BEEN REMOVED FROM THE PACKAGE.
1 | #error THIS FILE HAS BEEN REMOVED FROM THE PACKAGE.
| ^~~~~
In file included from src/IORawData/SiPixelInputSources/interface/PixelSLinkDataInputSource.h:34:
poison/Utilities/StorageFactory/interface/StorageAccount.h:1:2: error: #error THIS FILE HAS BEEN REMOVED FROM THE PACKAGE.
1 | #error THIS FILE HAS BEEN REMOVED FROM THE PACKAGE.
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47135
Pull request #49594 was updated. @Alejandro1400, @Dr15Jones, @JanChyczynski, @Moanwar, @arunhep, @atpathak, @jfernan2, @lviliani, @makortel, @mandrenguyen, @mkirsano, @perrotta, @sensrcn, @smuzaffar, @srimanob, @theofil can you please check and sign again.
-1
Failed Tests: UnitTests
Size: This PR adds an extra 416KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49890/summary.html
COMMIT: 46231f2fa863a3059046c1db684f6d3de915fa71
CMSSW: CMSSW_16_0_X_2025-12-10-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49594/49890/install.sh to create a dev area with all the needed externals and cmssw changes.
The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
- @stahlleiton cms-sw/cmssw#49561
You can see more details here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49890/git-recent-commits.json https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49890/git-merge-result
Failed Unit Tests
I found 3 errors in the following unit tests:
---> test TestEdmFastMerge had ERRORS ---> test TestIOPoolCommonTP had ERRORS ---> test TestPoolInputOldFormat had ERRORS
Comparison Summary
Summary:
- You potentially added 4 lines to the logs
- ROOTFileChecks: Some differences in event products or their sizes found
- Reco comparison results: 1 differences found in the comparisons
- Reco comparison had 4 failed jobs
- DQMHistoTests: Total files compared: 53
- DQMHistoTests: Total histograms compared: 4273241
- DQMHistoTests: Total failures: 64
- DQMHistoTests: Total nulls: 0
- DQMHistoTests: Total successes: 4273157
- DQMHistoTests: Total skipped: 20
- DQMHistoTests: Total Missing objects: 0
- DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
- Checked 227 log files, 198 edm output root files, 53 DQM output files
- TriggerResults: no differences found
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47136
Pull request #49594 was updated. @Alejandro1400, @Dr15Jones, @JanChyczynski, @Moanwar, @arunhep, @atpathak, @cmsbuild, @jfernan2, @lviliani, @makortel, @mandrenguyen, @mkirsano, @perrotta, @sensrcn, @smuzaffar, @srimanob, @theofil can you please check and sign again.
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49594/47137
Pull request #49594 was updated. @Alejandro1400, @Dr15Jones, @JanChyczynski, @Moanwar, @arunhep, @atpathak, @cmsbuild, @jfernan2, @lviliani, @makortel, @mandrenguyen, @mkirsano, @perrotta, @sensrcn, @smuzaffar, @srimanob, @theofil can you please check and sign again.
-1
Failed Tests: UnitTests
Size: This PR adds an extra 416KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49891/summary.html
COMMIT: 88378e096d6252571e5e58fdfbf8ec4c06c0cf62
CMSSW: CMSSW_16_0_X_2025-12-10-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49594/49891/install.sh to create a dev area with all the needed externals and cmssw changes.
The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
- @stahlleiton cms-sw/cmssw#49561
You can see more details here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49891/git-recent-commits.json https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dc70c/49891/git-merge-result
Failed Unit Tests
I found 1 errors in the following unit tests:
---> test TestIOPoolCommonTP had ERRORS
Comparison Summary
Summary:
- You potentially added 5 lines to the logs
- ROOTFileChecks: Some differences in event products or their sizes found
- Reco comparison results: 4 differences found in the comparisons
- Reco comparison had 4 failed jobs
- DQMHistoTests: Total files compared: 53
- DQMHistoTests: Total histograms compared: 4273241
- DQMHistoTests: Total failures: 27
- DQMHistoTests: Total nulls: 0
- DQMHistoTests: Total successes: 4273194
- DQMHistoTests: Total skipped: 20
- DQMHistoTests: Total Missing objects: 0
- DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
- Checked 227 log files, 198 edm output root files, 53 DQM output files
- TriggerResults: no differences found
The unit test failure was expected and will go away once this PR is merged into the IBs. That is because the problem is ROOT is seeing the 'same' plugin in both the release area and the work area because the plugin was moved to a new shared library.
@cms-sw/generators-l2 @cms-sw/alca-l2 please sign. The changes were simply to move files to a new directory.
@cms-sw/generators-l2 @cms-sw/alca-l2 please sign. The test failure is expected and will go away once this is in the IB.
@cms-sw/generators-l2 please sign
@cms-sw/orp-l2 please consider continuing without generator signature as the changes related to generator were simply to change where an include comes from.