cmssw icon indicating copy to clipboard operation
cmssw copied to clipboard
verified publisher icon cms-sw

Trivial serialisation plugin mechanism

Open fwyzard opened this issue 1 month ago • 27 comments

PR description:

Introduce ngt::SerialiserFactory, a plugin-based trivial serialiser, and the serialisation plugins for all PortableHostCollection- and PortableHostObject-derived data formats.

This relies on some new components:

  • ngt::AnyBuffer: ngt::AnyBuffer behaves like std::any, with two differences: it can only be used with trivially copyable types, and provides access to the underlying memory buffer to allow memcpy'ing its content.
  • the ngt::MemoryCopyTraits<T> type traits. The struct ngt::MemoryCopyTraits<T> should be specialised for the types that can be safely memcpy'ed.

And is accompanied by various tests:

  • Catch2 tests for the ngt::MemoryCopyTraits.
  • Catch2 tests for the ngt::SerialiserFactory.
  • integration tasts based on the ngt::GenericCloner.

The ngt::GenericCloner is included to demonstrate how the new functionalities can be used, and to implement various integration tests. This EDProducer will clone all the event products declared in its configuration, using either the plugin-based NGT trivial serialisation, or the products' ROOT dictionaries.

PR tests:

The new unit tests pass.

fwyzard avatar Nov 25 '25 21:11 fwyzard

cms-bot internal usage

cmsbuild avatar Nov 25 '25 21:11 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/46953

  • There are other open Pull requests which might conflict with changes you have proposed:
    • File DataFormats/Portable/BuildFile.xml modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): #32632, #49072, #49152, #49233
    • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): #32632, #49152, #49233, #49293
    • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): #49152
    • File DataFormats/SiPixelClusterSoA/BuildFile.xml modified in PR(s): #49471
    • File FWCore/Framework/interface/WrapperBaseHandle.h modified in PR(s): #32632, #49152, #49233, #49449
    • File FWCore/Framework/interface/WrapperBaseOrphanHandle.h modified in PR(s): #32632, #49152, #49233, #49449
    • File FWCore/Framework/src/WrapperBaseHandle.cc modified in PR(s): #32632, #49152, #49233, #49449
    • File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): #32632, #49233, #49449
    • File FWCore/Integration/plugins/WrapperBaseProducer.cc modified in PR(s): #32632, #49233, #49449
    • File FWCore/Integration/test/BuildFile.xml modified in PR(s): #32632, #49233, #49449
    • File FWCore/Integration/test/testWrapperBaseProducer_cfg.py modified in PR(s): #32632, #49233, #49449

cmsbuild avatar Nov 25 '25 21:11 cmsbuild

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

  • DataFormats/BeamSpot (reconstruction)
  • DataFormats/EcalDigi (simulation)
  • DataFormats/EcalRecHit (reconstruction)
  • DataFormats/HGCalDigi (simulation)
  • DataFormats/HGCalReco (reconstruction)
  • DataFormats/HcalDigi (simulation)
  • DataFormats/HcalRecHit (reconstruction)
  • DataFormats/ParticleFlowReco (reconstruction)
  • DataFormats/Portable (heterogeneous)
  • DataFormats/PortableTestObjects (heterogeneous)
  • DataFormats/SiPixelClusterSoA (heterogeneous, reconstruction)
  • DataFormats/SiPixelDigiSoA (heterogeneous, reconstruction)
  • DataFormats/TrackSoA (heterogeneous, reconstruction)
  • DataFormats/TrackingRecHitSoA (heterogeneous, reconstruction)
  • DataFormats/VertexSoA (heterogeneous, reconstruction)
  • FWCore/Framework (core)
  • FWCore/Integration (core)
  • HeterogeneousCore/SerialisationCore (****)

The following packages do not have a category, yet:

HeterogeneousCore/SerialisationCore Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@Dr15Jones, @civanch, @cmsbuild, @fwyzard, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @smuzaffar, @srimanob can you please review it and eventually sign? Thanks. @IzaakWN, @ReyerBand, @VinInn, @VourMa, @abdoulline, @argiro, @bsunanda, @dkotlins, @elusian, @ferencek, @gpetruc, @hatakeyamak, @lgray, @makortel, @mariadalfonso, @missirol, @mmasciov, @mmusich, @mroguljic, @mtosi, @pfs, @rchatter, @rovere, @thomreis, @tsusa, @wang0jin, @wddgit this is something you requested to watch as well. @ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild avatar Nov 25 '25 21:11 cmsbuild

please test

fwyzard avatar Nov 25 '25 22:11 fwyzard

type ngt

fwyzard avatar Nov 25 '25 22:11 fwyzard

@ghyls FYI

fwyzard avatar Nov 25 '25 22:11 fwyzard

+1

Size: This PR adds an extra 100KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ef6e7/49681/summary.html COMMIT: 63bc511482c767a961010ac59a3aec0b9bf1acb2 CMSSW: CMSSW_16_0_X_2025-11-25-1100/el8_amd64_gcc13 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49475/49681/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 2 lines to the logs
  • Reco comparison results: 3 differences found in the comparisons
  • Reco comparison had 4 failed jobs
  • DQMHistoTests: Total files compared: 55
  • DQMHistoTests: Total histograms compared: 4503038
  • DQMHistoTests: Total failures: 122
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4502896
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
  • Checked 235 log files, 208 edm output root files, 55 DQM output files
  • TriggerResults: no differences found

cmsbuild avatar Nov 26 '25 02:11 cmsbuild

Rebased to the current master.

fwyzard avatar Nov 26 '25 13:11 fwyzard

please test

fwyzard avatar Nov 26 '25 13:11 fwyzard

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/46965

  • There are other open Pull requests which might conflict with changes you have proposed:
    • File DataFormats/Portable/BuildFile.xml modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): #32632, #49072, #49152, #49233
    • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): #32632, #49152, #49233, #49293
    • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): #49152

cmsbuild avatar Nov 26 '25 13:11 cmsbuild

Pull request #49475 was updated. @civanch, @fwyzard, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @srimanob can you please check and sign again.

cmsbuild avatar Nov 26 '25 13:11 cmsbuild

+1

Size: This PR adds an extra 80KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ef6e7/49690/summary.html COMMIT: 5c7cf7aaf156ca2b97d479a06cf6b1a8c5117a4b CMSSW: CMSSW_16_0_X_2025-11-25-2300/el8_amd64_gcc13 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49475/49690/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

  • @fwyzard cms-sw/cmssw#49449
  • @fwyzard cms-sw/cmssw#49110
  • @smuzaffar cms-sw/cmssw#49472
  • @fwyzard cms-sw/cmssw#49471
  • @bsunanda cms-sw/cmssw#49478

You can see more details here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ef6e7/49690/git-recent-commits.json https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ef6e7/49690/git-merge-result

Comparison Summary

Summary:

  • You potentially added 1 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 11 differences found in the comparisons
  • Reco comparison had 4 failed jobs
  • DQMHistoTests: Total files compared: 55
  • DQMHistoTests: Total histograms compared: 4502606
  • DQMHistoTests: Total failures: 21
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4502565
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
  • Checked 235 log files, 208 edm output root files, 55 DQM output files
  • TriggerResults: no differences found

cmsbuild avatar Nov 26 '25 18:11 cmsbuild

+1

civanch avatar Nov 27 '25 08:11 civanch

+1

jfernan2 avatar Nov 27 '25 14:11 jfernan2

Rebased on top of CMSSW_16_0_0_pre3.

fwyzard avatar Nov 28 '25 23:11 fwyzard

please test

fwyzard avatar Nov 28 '25 23:11 fwyzard

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/47013

  • There are other open Pull requests which might conflict with changes you have proposed:
    • File DataFormats/Portable/BuildFile.xml modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): #32632, #49072, #49152, #49233
    • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): #32632, #49152, #49233, #49293
    • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): #49152
    • File HeterogeneousCore/SerialisationCore/README.md modified in PR(s): #49235
    • File HeterogeneousCore/SerialisationCore/interface/AnyBuffer.h modified in PR(s): #49235

cmsbuild avatar Nov 28 '25 23:11 cmsbuild

Pull request #49475 was updated.

cmsbuild avatar Nov 28 '25 23:11 cmsbuild

+1

Size: This PR adds an extra 84KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ef6e7/49735/summary.html COMMIT: b1152115bf67197f424db5e3635f80040423ed26 CMSSW: CMSSW_16_0_X_2025-11-28-2300/el8_amd64_gcc13 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49475/49735/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 3 differences found in the comparisons
  • Reco comparison had 4 failed jobs
  • DQMHistoTests: Total files compared: 55
  • DQMHistoTests: Total histograms compared: 4502606
  • DQMHistoTests: Total failures: 67
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4502519
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
  • Checked 235 log files, 208 edm output root files, 55 DQM output files
  • TriggerResults: no differences found

cmsbuild avatar Nov 29 '25 03:11 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/47014

  • There are other open Pull requests which might conflict with changes you have proposed:
    • File DataFormats/Portable/BuildFile.xml modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): #32632, #49072, #49152, #49233
    • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): #32632, #49152, #49233, #49293
    • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): #49152
    • File HeterogeneousCore/SerialisationCore/README.md modified in PR(s): #49235
    • File HeterogeneousCore/SerialisationCore/interface/AnyBuffer.h modified in PR(s): #49235

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

Pull request #49475 was updated.

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/47015

  • There are other open Pull requests which might conflict with changes you have proposed:
    • File DataFormats/Portable/BuildFile.xml modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): #32632, #49072, #49152, #49233
    • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): #32632, #49152, #49233, #49293
    • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): #49152
    • File HeterogeneousCore/SerialisationCore/README.md modified in PR(s): #49235
    • File HeterogeneousCore/SerialisationCore/interface/AnyBuffer.h modified in PR(s): #49235

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

Pull request #49475 was updated.

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/47016

  • There are other open Pull requests which might conflict with changes you have proposed:
    • File DataFormats/Portable/BuildFile.xml modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): #32632, #49072, #49152, #49233
    • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): #32632, #49152, #49233, #49293
    • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): #49152
    • File HeterogeneousCore/SerialisationCore/README.md modified in PR(s): #49235
    • File HeterogeneousCore/SerialisationCore/interface/AnyBuffer.h modified in PR(s): #49235

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

Pull request #49475 was updated.

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/47017

  • There are other open Pull requests which might conflict with changes you have proposed:
    • File DataFormats/Portable/BuildFile.xml modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): #32632, #49233
    • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): #49072
    • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): #32632, #49072, #49152, #49233
    • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): #32632, #49152, #49233, #49293
    • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): #49152
    • File HeterogeneousCore/SerialisationCore/README.md modified in PR(s): #49235
    • File HeterogeneousCore/SerialisationCore/interface/AnyBuffer.h modified in PR(s): #49235

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

Pull request #49475 was updated.

cmsbuild avatar Nov 29 '25 07:11 cmsbuild

To do:

  • [x] move HeterogeneousCore/SerialisationCore/interface/MemoryCopyTraits.h to DataFormats/TrivialSerialisation to remove the dependency of the other DataFormats on ``HeterogeneousCore/SerialisationCore(which depends onFWCore`)
  • [x] rename HeterogeneousCore/SerialisationCore to HeterogeneousCore/TrivialSerialisation
  • [x] rename 'serialisation.cc' to 'TrivialSerialisation.cc'

fwyzard avatar Dec 16 '25 17:12 fwyzard

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/47192

ERROR: Build errors found during clang-tidy run.

src/DataFormats/SiPixelClusterSoA/interface/SiPixelClustersHost.h:10:10: error: 'HeterogeneousCore/TrivialSerialisation/interface/MemoryCopyTraits.h' file not found [clang-diagnostic-error]
   10 | #include "HeterogeneousCore/TrivialSerialisation/interface/MemoryCopyTraits.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Suppressed 583 warnings (583 in non-user code).
--
src/DataFormats/SiPixelDigiSoA/interface/SiPixelDigiErrorsHost.h:14:10: error: 'HeterogeneousCore/TrivialSerialisation/interface/MemoryCopyTraits.h' file not found [clang-diagnostic-error]
   14 | #include "HeterogeneousCore/TrivialSerialisation/interface/MemoryCopyTraits.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Suppressed 583 warnings (583 in non-user code).
--
src/DataFormats/SiPixelClusterSoA/interface/SiPixelClustersHost.h:10:10: error: 'HeterogeneousCore/TrivialSerialisation/interface/MemoryCopyTraits.h' file not found [clang-diagnostic-error]
   10 | #include "HeterogeneousCore/TrivialSerialisation/interface/MemoryCopyTraits.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Suppressed 583 warnings (583 in non-user code).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild avatar Dec 16 '25 22:12 cmsbuild

Rebased to the current master, and updated the development following the discussion at the Core Software meeting.

In particular, DataFormats/Portable now depends only on DataFormats/Common and DataFormats/TrivialSerialisation, and the latter does not have any dependencies.

fwyzard avatar Dec 16 '25 22:12 fwyzard