data-curation
data-curation copied to clipboard
cernopendata
Data ingestion and curation tools
Hi @tiborsimko , all, This is a very early attempt at adding the scripts in that are necessary for generating the Open Data Portal records for the ATLAS Open Data...
The cross-section utility does not seem to work for HiggsPhysics as under MC2015/HiggsPhysics in https://cernbox.cern.ch/files/link/public/EHpyrdJet939vGy It worked without problem for the StandardModelPhysics cases but for ``` "categories": { "primary": "Higgs...
Addresses #182 Adds code for all steps. The logic has been changed to find the provenance through the production chain. Input files are for testing only. Tested on 3 datasets...
Checklist for the 2016 data release. This is to start the preparations. See also the volume estimates in - https://indico.cern.ch/event/1029215/contributions/4324588/attachments/2234005/3786244/DPOA_news_27_04_2021.pdf#page=5, and - https://indico.cern.ch/event/1192611/contributions/5013791/attachments/2512351/4318700/CERN_ODWG_CMS_Sept_2022.pdf - [x] collision dataset list, see #155...
(from #124) ## JetHT for testing Transfer started for Run2016G JetHT ``` $ echo $dataset /JetHT/Run2016G-UL2016_MiniAODv2-v2/MINIAOD $ rucio add-rule cms:$dataset 1 T3_CH_CERN_OpenData 5aa90ff16f3541659de09f8406702366 ``` ``` -bash-4.2$ echo $dataset /JetHT/Run2016G-UL2016_MiniAODv2_NanoAODv9-v1/NANOAOD -bash-4.2$...
for https://github.com/cernopendata/opendata.cern.ch/issues/3569 Write a script to get the pile-up record json. Single production step (process neutrino particle gun = nothing) see [McM](https://cms-pdmv-prod.web.cern.ch/mcm/requests?prepid=PPD-RunIISummer20ULPrePremix-00003&page=0&shown=140737488355327) Sampling pileup from the GEN-SIM pile-up sample [/MinBias_TuneCP5_13TeV-pythia8/RunIISummer20UL16SIM-106X_mcRun2_asymptotic_v13-v2/GEN-SIM](https://cmsweb.cern.ch/das/request?input=/MinBias_TuneCP5_13TeV-pythia8/RunIISummer20UL16SIM-106X_mcRun2_asymptotic_v13-v2/GEN-SIM)...
Write a script to create new event display records An example of an existing event display record: - https://opendata-qa.cern.ch/record/7144 The json description is in - https://github.com/cernopendata/opendata.cern.ch/blob/master/cernopendata/modules/fixtures/data/records/cms-eventdisplay-files-Run2012C.json See https://github.com/cernopendata/opendata.cern.ch/issues/3200
(from #124) Transfer a part (max 100 TB) of the pileup premix dataset [/Neutrino_E-10_gun/RunIISummer20ULPrePremix-UL16_106X_mcRun2_asymptotic_v13-v1/PREMIX](https://cmsweb.cern.ch/das/request?input=dataset%3D%2FNeutrino_E-10_gun%2FRunIISummer20ULPrePremix-UL16_106X_mcRun2_asymptotic_v13-v1%2FPREMIX&instance=prod/global) (see https://cms-opendata-releaseguide.docs.cern.ch/data_to_be_released/pileup_dataset/)
(from #124) Test the GT reading - [x] from the local db in the `gitlab-registry.cern.ch/cms-cloud/cmssw-docker-opendata/cmssw_10_6_30-slc7_amd64_gcc700` container - [x] from /cvmfs with a cvmfs mount of /cvmfs/cms-opendata-conddb.cern.ch/ - [x] from /cvmfs...