WMCore
WMCore copied to clipboard
dmwm
Increasing Number of workflows with Duplicates Lumis
Impact of the bug Duplicate Lumis in output files affect the Output Datasets, The workflows with duplicates in their lumis are not announced automatically and need Manual Operations from P&R to remove the files with duplicate lumis.
Describe the bug There has been an increase in Workflows with duplicates in their Outputs over the past weeks.
Monitoring Link: https://monit-grafana.cern.ch/goto/W0feXBxSR?orgId=11
It has also affected RelVal workflows, as described in this ticket https://its.cern.ch/jira/browse/CMSPROD-165
A recent example of a workflow with duplicates
- https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_BPH-Run3Summer23BPixGS-00113
- https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_GEN-Run3Summer23wmLHEGS-00468
For this workflow I have invalidated the files in DBS with duplicate Lumis
- https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_GEN-Run3Summer23wmLHEGS-00465
How to reproduce it I can try submitting one of the above workflows as a backfill and see if its output also has duplicates.
Expected behavior Output Datasets to not have files with duplicate Lumis.
Thank you @amaltaro for the endpoint to check workflows by their output datasets. I checked all the workflows with duplicates and multiple Relval workflows had the same output datasets.
However, the following workflows do not have such issues and I have also checked their input datasets for duplicates, They do not have any duplicate lumis in their input dataset either.
cmsunified_task_BPH-Run3Summer23BPixGS-00113__v1_T_240324_130707_6996 cmsunified_task_BPH-Run3Summer23GS-00107__v1_T_240115_172538_9295 cmsunified_task_GEN-Run3Summer23BPixwmLHEGS-00417__v1_T_240324_130542_5642 cmsunified_task_GEN-Run3Summer23BPixwmLHEGS-00428__v1_T_240324_121816_7540 cmsunified_task_GEN-Run3Summer23wmLHEGS-00438__v1_T_240324_123619_5453 cmsunified_task_GEN-Run3Summer23wmLHEGS-00451__v1_T_240324_130421_9550 cmsunified_task_GEN-Run3Summer23wmLHEGS-00464__v1_T_240324_125830_5588 cmsunified_task_GEN-Run3Summer23wmLHEGS-00468__v1_T_240324_130100_3586 cmsunified_task_HIG-Run3Summer22EEwmLHEGS-00425__v1_T_240123_063355_4488 cmsunified_task_HIG-Run3Summer22EEwmLHEGS-01105__v1_T_240328_105719_1779 cmsunified_task_HIG-Run3Summer22wmLHEGS-00749__v1_T_240115_065425_3703
@hassan11196 Hassan, I am not saying I am going to debug it :) But just to help WM on debugging it, would you have further details for one of these workflows? Which dataset? Which run/lumi and file have dups?
@hassan11196 Hassan, I was looking for a workflow with very few stats, and I found this cmsunified_task_EXO-Run3Summer22MiniAODv4-00662__v1_T_240213_134751_3377 which has been sitting in completed status since 12/Feb/2024.
I implemented lumi check at 3 levels: dataset, block and file; and indeed they differ as can be seen in the following table:
where we can see 4 duplicate lumis in the input dataset. In case you want to check on your side, here is the duplicate report for the input dataset (which can potentially cause duplicate lumis in the output, and it did!):
Lumi {7} is common between /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/f956c9e4-c5ae-4ec6-8c77-71bfc3002cf9.root and /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/5d37dc29-67f7-41fb-8a02-4f4645464938.root
Lumi {18} is common between /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/9cf8c597-1746-4709-8d90-e0286755ec42.root and /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/8acd7ad5-d887-4a0d-9a94-5b435149a2d6.root
Lumi {15} is common between /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/9cf8c597-1746-4709-8d90-e0286755ec42.root and /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/70ba64c0-3cb9-4469-9ba6-5e3c0b28612e.root
Lumi {4} is common between /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/1c9e3f46-8505-45ed-bea2-4e9e84b1a827.root and /store/mc/Run3Summer22DRPremix/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/AODSIM/124X_mcRun3_2022_realistic_v12-v2/2810000/475ae403-5f5f-4d51-aa6d-f4f3f308d91b.root
lastly, this dataset above was (potentially) produced by 3 workflows, according to request reqmgr2 api.
Just for your information, the distinction of those 3 different ways to calculate lumis in the dataset are:
- for
dataset: queryfilesummariesfor a dataset and valid files only. - for
block: queryfilesummariesfor each block in the dataset and with valid files only. - for
file: queryfilelumisfor each block in the dataset and with valid files only.
If you want, I can polish my python notebook and share it with you tomorrow, such that you can check the other workflows.
PS.: the workflow that produced the dataset above did not have any input dataset. So duplication originated on the direct parent workflow.
Hi @amaltaro, Thank you for this investigation. I did not update the ticket but I was able to find the files that that had duplicated lumis using our scripts but was not able to find the lumis yet, Please do share your notebook (even in unpolished state) this will help me a lot.
We now have this tool in our dashboard which we use to find the files to invalidate for duplicate lumis. https://pnr-dev-hassan11196.app.cern.ch/tools#:~:text=Check%20For%20Duplicates%20in%20Workflow
I will provide you an updated list of wfs that had duplicated lumis, along with list of files and lumi run numbers.
for dataset: query `filesummaries` for a dataset and valid files only.
for block: query `filesummaries` for each block in the dataset and with valid files only.
You mentioned the same query for block and dataset. So inside each the filesummaries for a dataset, their are filesummaries for each block?
Another question I have is, if multiple workflows are writing to a dataset, what are the parameters that control the workflow outputs what lumi no's. I assume FirstLumi and LumiList.
If its explained somewhere in the doc, just let me know which keywords to search.
Lastly, I want to minimize your your time with trivial things given wmcores priorities this quarter, you can just give me pointers to find the stuff, I will gather all logs and relevant info for you in a single place and make it easy for you to narrow down the issue.
Thanks a lot Alan.
@hassan11196 Hassan, I think we covered most of this over zoom today, but please let me know if anything needs follow up.
Another question I have is, if multiple workflows are writing to a dataset, what are the parameters that control the workflow outputs what lumi no's. I assume FirstLumi and LumiList.
If you planned on having multiple workflows writing to the same output dataset, then yes, you need to use FirstLumi and FirstEvent to avoid getting "duplicates. Here is a JSON workflow template that exemplifies its usage.
However, most - if not all - of these duplicate lumis happen unintentionally, and you can either invalidate given files in DBS or recreate the output dataset in a v++ setup.
We briefly discussed the notebook today, and here it is: https://github.com/amaltaro/ProductionTools/blob/master/dbs_dup_lumis.ipynb note that the workflow name is currently hard-wired in the notebook (at the top).
I am moving this issue to waiting, once we confirm with another workflow or two that there is no problem on the WM side, I would suggest to get it closed. Please let us know how it goes. Thanks!
Hello @amaltaro I have compiled details about workflows currently in the system with duplicates. Most workflows don't have duplicates in input, No ACDC wfs and no multiple wfs writing to the same dataset. On Average, only 1 or 2 lumis are duplicated. Please have a look and let me know what else should I look into.
https://docs.google.com/document/d/1bH6etTBucsw5F_wUKiqHBHuN1fRAlLUuGEbAaGAwd2g/edit?usp=sharing
Thank you.
Hi @amaltaro As an extension to the above list, I have noticed duplicate lumis in another workflow. https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=ReReco-Run2023C-Muon1-22Sep2023_v4-00001
For background context https://its.cern.ch/jira/browse/CMSPROD-172
We reproduced this dataset as V2, but there is a discrepancy between MiniAOD and NanoAOD events and lumis. NanoAOD has more lumis than MiniAOD.
lumi completion 36748 expected 36822 for /Muon1/Run2023C-22Sep2023_v4-v2/MINIAOD
lumi completion 36818 expected 36822 for /Muon1/Run2023C-22Sep2023_v4-v2/NANOAOD
cmsunified_Run2023C_Muon1_22Sep2023_v4_240425_131830_8336 is not completed.
Current stats:
{
"/Muon1/Run2023C-22Sep2023_v4-v2/MINIAOD": 0.963855421686747,
"/Muon1/Run2023C-22Sep2023_v4-v2/NANOAOD": 0.9761904761904762
}
Summary for: /Muon1/Run2023C-PromptReco-v4/AOD
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 84 | 84 | 84 |
| num_file | 19268 | 19268 | 19268 |
| num_event | 138917988 | 138917988 | 138917988 |
| num_lumi | 36822 | 36822 | 36822 |
Summary for: /Muon1/Run2023C-22Sep2023_v4-v2/MINIAOD
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 11 | 11 | 11 |
| num_file | 2140 | 2140 | 2140 |
| num_event | 138680345 | 138680345 | 138680345 |
| num_lumi | 36748 | 36748 | 36748 |
Summary for: /Muon1/Run2023C-22Sep2023_v4-v2/NANOAOD
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 10 | 10 | 10 |
| num_file | 106 | 106 | 106 |
| num_event | 138894942 | 138894942 | 0 |
| num_lumi | 36818 | 36818 | 36818 |
our initial suspicion was that it might be caused by duplicated lumis. However, our duplication check in Unified did not detect any issues.
I tried the getFileLumis method in your notebook and discovered that the output files indeed did have duplicated lumis but since they were only in the individual file and not across files Our duplicate check did not detect them.
Here is a complete list of duplicated Lumis,
/Muon1/Run2023C-22Sep2023_v4-v2/MINIAOD
1) lfn 2820001/99d5cc32-2fa7-4713-884f-1d3361e848d3.root has lumi [41] duplicated
2) lfn 2820002/b1ebe6fa-70a0-4f62-9974-b48628a28808.root has lumi [55, 56] duplicated
3) lfn 2820001/bb2ad438-d997-42f4-8334-318535170983.root has lumi [223] duplicated
4) lfn 2820001/ff4ffb73-a311-4b39-bdf6-58fc647f43e3.root has lumi [59, 62] duplicated
5) lfn 2820002/d9869131-ceac-49a8-8702-9a758e47fe59.root has lumi [122] duplicated
6) lfn 2820001/e09b9864-caac-430a-a44a-433b9dfe1144.root has lumi [242] duplicated
7) lfn 2820001/786b47c1-4d3d-4f8e-afd3-a9db0433ff22.root has lumi [14] duplicated
8) lfn 2820002/0cbf95a6-822a-41ca-b92f-b2ac09d8447a.root has lumi [173] duplicated
9) lfn 2820001/e56ed56d-d4fa-45b1-adbf-d4b1efa70f07.root has lumi [208] duplicated
10) lfn 2820001/055620fe-6831-407a-b7da-a8c4fd09f9e8.root has lumi [143, 140, 139] duplicated
11) lfn 2820001/9c3f95f8-eb0c-4de0-83ed-95394fec722e.root has lumi [154, 152] duplicated
12) lfn 2820000/ca6f5a26-1e5e-498d-acba-407f65e0e4a4.root has lumi [87] duplicated
13) lfn 2820000/52910047-4650-4589-92a3-71482c9b2563.root has lumi [41] duplicated
14) lfn 2820000/fb216061-e3e8-4fab-be63-e5dc5b2324a5.root has lumi [78] duplicated
15) lfn 2820001/76e15efb-595e-426d-b231-356cd9fe2c43.root has lumi [190, 228] duplicated
16) lfn 2820000/25d31b3d-9099-46f6-881d-f8e4b56d6978.root has lumi [116, 114, 120, 119, 115] duplicated
17) lfn 2820000/0b606290-d6a9-42f6-8809-e111df637e7f.root has lumi [96, 98, 94, 95, 97] duplicated
18) lfn 2820000/2614cb91-55a9-4be3-a723-202971f84497.root has lumi [9, 7, 8, 6, 16] duplicated
19) lfn 2820001/beceb2f9-9249-43b1-8ec0-94962a307bfe.root has lumi [607] duplicated
20) lfn 2820001/56a44855-7e4d-4721-b564-86682246f95c.root has lumi [224] duplicated
21) lfn 2820000/2eaf6551-a256-4794-8934-bb0c59a0fbea.root has lumi [43, 46, 45, 44] duplicated
22) lfn 2820001/c1f46a94-9fda-463f-8fbb-a09761861c35.root has lumi [217] duplicated
23) lfn 2820000/d6ac376a-3912-40bb-a3cd-4a7701634d14.root has lumi [12, 13, 66, 4, 5, 7, 10, 11, 6, 22, 24, 25, 3, 1, 17, 9, 21, 30, 14, 27, 29, 2, 15, 18, 23, 16, 28, 8, 19, 20, 26] duplicated
24) lfn 2820001/5e11e894-da68-4edc-b729-52a5eb356728.root has lumi [380] duplicated
25) lfn 2820000/f9cb4995-da4a-4054-a03a-03dde37570ad.root has lumi [580] duplicated
26) lfn 2820001/de123cd8-2000-47a7-856e-84b27eb32d2c.root has lumi [64, 66, 67, 65] duplicated
27) lfn 2820001/1b3e4f7d-21d2-4ea6-8b25-288b95b68751.root has lumi [433, 435] duplicated
28) lfn 2820000/dd15345c-4662-4bba-9da1-e5a90f967b2e.root has lumi [2, 17, 22, 7, 14, 19, 21, 23, 28, 30, 35, 44, 9, 15, 24, 4, 18, 27, 34, 93, 1, 29, 33, 36, 3, 5, 20, 26, 32, 98, 6, 10, 11, 16, 8, 25, 13, 61, 12, 31, 76] duplicated
29) lfn 2820000/c4c6735e-8ac4-4242-be4c-b243b39be652.root has lumi [124] duplicated
30) lfn 2820000/d48daa74-5ae3-4299-bdd2-ee499ac5bf54.root has lumi [38, 34, 31, 39, 30, 35, 32] duplicated
31) lfn 2820000/cd7d0ab8-035d-40e1-a384-b1a3541d485b.root has lumi [805] duplicated
32) lfn 2820000/96ef6241-35f4-4abc-8b02-8c6b1c114ed3.root has lumi [35, 7, 36, 6, 8, 9] duplicated
33) lfn 2820000/1602f71b-458d-44a3-afef-fc087e1373af.root has lumi [210] duplicated
34) lfn 2820000/e047565b-19a3-4831-b4aa-3e25f63afd7b.root has lumi [162] duplicated
35) lfn 2820000/83cc2233-dd08-492e-a0e9-fb7901341fca.root has lumi [11] duplicated
36) lfn 2820001/66b8d1bd-26f3-42b4-ba86-70c7cbb98568.root has lumi [992] duplicated
37) lfn 2820000/f2682d13-4146-48ce-91c3-48c733f06d73.root has lumi [113, 118, 111, 112, 114, 116, 115, 117, 119] duplicated
38) lfn 2820000/87c5fa61-245a-490c-b514-5f5883a7cb9d.root has lumi [3, 8, 2, 4, 5, 1, 11, 6, 9, 67, 64, 7, 10, 21, 39] duplicated
39) lfn 2820000/9cbb0011-168d-4c6f-b509-15eb25c258a9.root has lumi [72, 73] duplicated
40) lfn 2820000/7ddd9326-9e16-4055-87f6-c5d247d00a79.root has lumi [1054] duplicated
41) lfn 2820000/2c7ceab4-b366-4815-9640-4c0d80272d04.root has lumi [3, 1, 2, 5, 12, 14, 4, 8, 11, 9, 13, 7, 17, 16, 10, 15, 6] duplicated
42) lfn 2820001/6e071d2d-e79a-4fe5-b6d4-e66fee150e26.root has lumi [94, 100] duplicated
43) lfn 2820000/77bb1c27-040a-4e8e-857b-d2ea56cacc6f.root has lumi [86] duplicated
44) lfn 2820000/94021fec-8cf1-40ed-9f9a-9b4b275a1215.root has lumi [5, 9, 8, 11, 6, 48, 49, 4, 26] duplicated
45) lfn 2820000/3e252f74-48bf-4060-a727-9b67410395fc.root has lumi [51, 52, 103, 53, 50, 54] duplicated
46) lfn 2820000/0a386ad9-caf0-4799-b002-a588ddd2fb6e.root has lumi [141] duplicated
47) lfn 2820000/7df65b90-cd9c-469b-b053-c07a0b067350.root has lumi [60, 63, 50, 59, 87, 104, 61, 54, 56, 62, 53, 77, 37, 52, 55, 57, 78, 51, 58, 79] duplicated
48) lfn 2820000/ef5b56ed-13ac-4d4a-9ea5-048891fde444.root has lumi [2, 1, 3, 14, 13, 7, 8, 19, 20] duplicated
49) lfn 2820000/1f4b58f6-a112-40ad-9ddd-627b383e799a.root has lumi [71] duplicated
50) lfn 2820000/43b50dc1-7de1-476a-92ef-eda51b472ad4.root has lumi [19, 20] duplicated
51) lfn 2820000/86ccff2a-b658-45fc-a79d-711a239c161b.root has lumi [106, 95] duplicated
52) lfn 2820000/adffc106-af45-4f3f-82f7-89a64ba20ae9.root has lumi [129, 130, 126] duplicated
53) lfn 2820000/008bb8d9-09a0-4ced-8575-c111ae6406a6.root has lumi [51] duplicated
54) lfn 2820000/5fd67233-c729-4be3-8f92-9d092f0b3d99.root has lumi [173] duplicated
55) lfn 2820001/62592097-f39f-4f34-9e32-aeba1c9e86b1.root has lumi [10, 18, 11, 37, 38, 33, 19, 9, 32, 16] duplicated
56) lfn 2820000/e4eb2d30-e541-4927-a767-b9a945c9eb25.root has lumi [48, 49, 132, 131, 126] duplicated
57) lfn 2820000/ae92ce38-3624-4707-a0b5-d157690107e9.root has lumi [74, 75, 73, 76, 1, 80, 81] duplicated
58) lfn 2820001/512eb131-bcf0-44bb-aff9-38fd931ff7d4.root has lumi [259] duplicated
59) lfn 2820000/0d536bd3-7545-4cab-9868-892dc2deba9c.root has lumi [128, 129] duplicated
60) lfn 2820001/fb1603df-5b89-4c92-b4e4-b71e1d1b3787.root has lumi [51, 64] duplicated
61) lfn 2820000/956b473e-9203-4af6-9808-b03c233d9a5d.root has lumi [84] duplicated
62) lfn 2820000/30a5d0e6-3dbd-47e9-bb4a-e2740dc93a92.root has lumi [4, 10, 29, 63, 125, 127, 26, 30, 35, 41, 57, 1, 2, 24, 25, 50, 51, 59, 62, 64, 8, 23, 28, 31, 36, 43, 44, 52, 61, 5, 9, 19, 55, 56, 119, 3, 6, 18, 27, 39, 40, 47, 53, 54, 15, 33, 122, 129, 117, 16, 49, 17, 37, 48, 120, 7, 21, 45, 46, 13, 34, 38, 12, 14, 60, 11, 58, 32, 20, 22, 42, 121] duplicated
63) lfn 2820001/284e3c3d-7d18-46bf-a33b-548e6c7b0443.root has lumi [557] duplicated
64) lfn 2820000/4c87397f-176f-4895-a7be-1b59b907c944.root has lumi [171, 167] duplicated
65) lfn 2820000/e6c326be-09c7-4d7a-9f26-40491165cd08.root has lumi [20, 29, 11, 15, 18, 24, 12, 19, 28, 10, 25, 14, 13] duplicated
66) lfn 2820000/b6361a6b-05b9-4a1a-aaf8-590add057ec0.root has lumi [26, 22, 23, 27, 28] duplicated
67) lfn 2820000/7a945244-2d87-4e6b-9702-e7d14cf9b35f.root has lumi [37] duplicated
68) lfn 2820000/fd925d4c-ec23-41e5-a03e-12efac0db652.root has lumi [121] duplicated
69) lfn 2820000/b3714f9f-cf72-4555-99c8-b6d68904d4eb.root has lumi [75] duplicated
70) lfn 2820000/12919361-bc73-4a70-ae79-d7dfbc8d6c61.root has lumi [1252, 1253] duplicated
71) lfn 2820000/0f2f91fc-4c09-4034-816c-217cce8eee34.root has lumi [1169] duplicated
72) lfn 2820000/bef1381d-54d7-40c3-9f80-de38cebd8b3b.root has lumi [70] duplicated
/Muon1/Run2023C-22Sep2023_v4-v2/NANOAOD
1) lfn 2820000/cf830fae-8ceb-4945-b72b-dcfac4c36e92.root has lumi [846, 629, 335, 555, 16, 54, 57, 691, 245, 246, 341, 788, 425, 750, 795, 110, 231, 235, 15, 409, 410, 93, 783, 692, 417, 627, 333, 689, 863, 94, 97, 416, 406, 415, 557, 233, 52, 991, 418, 407, 76, 777, 782, 787, 847, 98, 106, 554, 561, 779, 780, 426, 334, 697] duplicated
2) lfn 2820000/c93e29b1-d2a9-4265-9633-5603cafba65d.root has lumi [513, 7, 69, 71, 72, 73, 65, 91, 130, 19, 21, 176, 94, 79, 76, 61, 87, 106, 20, 166, 5, 60, 101, 62, 90, 95, 115, 167, 181, 25, 572, 30, 108, 326, 29, 105, 114, 9, 74, 75, 36, 111, 172, 6, 1061, 178, 70, 11, 974, 174, 182, 191, 66, 63, 89, 175, 170, 107] duplicated
3) lfn 2820000/70e9e734-b7a4-428c-bc8e-5c8dade08332.root has lumi [389, 395, 397, 398, 47, 53, 63, 235, 237, 423, 36, 46, 52, 318, 375, 374, 426, 442, 444, 447, 59, 429, 54, 396, 399, 41, 319, 388, 44, 57, 226, 325, 424, 746, 156, 51, 377, 537, 42, 64, 445, 49, 321, 50, 376, 386, 55, 320, 370, 236, 48, 43, 45, 60, 669, 369, 271, 56, 58, 37, 387] duplicated
4) lfn 2820000/521ee26c-77c9-4838-994c-b4928cdf450f.root has lumi [85, 72, 135, 336, 605, 67, 68, 116, 73, 338, 74, 76, 71, 79, 82, 245, 81, 339, 611] duplicated
5) lfn 2820000/1fb83620-3df6-4e06-8bfa-5981f018742b.root has lumi [441, 440, 30, 43, 536, 804, 46, 31, 215, 840, 331, 439, 44, 105, 255, 263, 336, 291, 542, 816, 34, 15, 16, 171, 828, 175, 382, 42, 103, 109, 172, 817, 173, 213, 379, 37, 260, 254, 508, 33, 13, 176, 333, 212, 107, 541, 110, 392, 448, 214, 296] duplicated
6) lfn 2820000/589c117e-0648-448b-baa8-82b7ad75287f.root has lumi [351, 102, 648, 94, 499, 352, 500, 93, 95, 647, 580, 364, 493, 219, 96, 89, 71, 100, 97, 496, 357, 376, 313] duplicated
7) lfn 2820000/07817de9-b6b2-4f7c-87f0-ec2278a2d79e.root has lumi [367, 370, 502, 229, 86, 37, 38, 241, 343, 372, 562, 658, 150, 288, 176, 91, 70, 84, 108, 35, 422, 207, 428, 736, 69, 515, 251, 90, 301, 427, 83, 733, 286, 290, 36, 743, 151, 368, 285, 421, 166, 85, 167, 287, 516, 34] duplicated
8) lfn 2820000/fe5f1fef-cd5d-436e-a57f-f87f39aef85b.root has lumi [756, 757, 759, 66, 248, 762, 483, 64, 65, 112, 771, 514, 521, 115, 712, 751, 755, 309, 110, 775, 504, 774, 758, 760, 111, 764, 123, 67, 786, 439, 1076, 302, 992, 795] duplicated
9) lfn 2820000/d7121b84-d70c-4c60-976f-0474f7847034.root has lumi [429, 423, 302, 256, 262, 389, 393, 249, 255, 301, 382, 383, 428, 300, 254, 299, 303, 518, 873] duplicated
10) lfn 2820000/647a7427-f50c-494c-9405-055908a257a0.root has lumi [226, 361, 420, 230, 359, 362, 422, 232] duplicated
11) lfn 2820000/1c4ada1d-3ceb-4287-9846-578b902540f2.root has lumi [284, 285, 762] duplicated
12) lfn 2820000/5f0cb15c-a4ce-46db-84d5-44305b500df8.root has lumi [265, 168, 153, 172, 174, 468, 160, 326, 171, 204, 84, 142, 320, 328, 140, 365, 566, 492, 493, 454, 147, 329, 453, 87, 143, 151, 170, 159, 145, 217, 362, 289, 1047, 149, 346, 297, 491, 276, 308, 144, 219, 150, 83, 294, 141, 169, 494, 255, 455, 167] duplicated
13) lfn 2820000/a3166712-a909-46ed-997e-9835636d5d21.root has lumi [436, 739, 740, 419, 430, 735, 165, 159, 49, 738, 435, 433, 162, 163, 303, 554, 736, 432, 53, 46] duplicated
14) lfn 2820000/c8de7913-ed34-4992-9a20-04a9e360996d.root has lumi [427, 308, 414, 95, 378, 395, 177, 472, 225, 227, 192, 796, 205, 307, 330, 379, 18, 480, 191, 794, 375, 206, 178, 148, 306, 230, 199, 292, 213, 424, 396] duplicated
15) lfn 2820000/b210bdac-d2ae-4164-ab09-10f246f9eaaa.root has lumi [338, 496, 448, 761, 236, 265, 765, 208, 380, 495, 500, 272, 384, 386, 455, 506, 237, 152] duplicated
16) lfn 2820000/3152c165-8237-4043-9c9e-4d2813373f25.root has lumi [229, 720, 200, 202, 411, 543, 579, 723, 544, 576, 473, 234, 236, 545, 718, 233, 577, 580, 571, 410, 715, 207] duplicated
17) lfn 2820000/ae481bcc-5ed8-4adb-ac7e-d3b7a37f7e8d.root has lumi [56, 55, 459, 31] duplicated
18) lfn 2820000/f1b2bac4-c269-4b79-8207-7a5632a18622.root has lumi [273, 281, 311, 321, 325, 336, 48, 256, 262, 10, 50, 53, 54, 57, 59, 76, 82, 1, 3, 5, 7, 12, 15, 16, 17, 6, 883, 128, 129, 137, 2, 31, 47, 52, 9, 41, 63, 8, 14, 25, 81, 96, 320, 13, 20, 33, 44, 45, 64, 105, 67, 55, 79, 11, 4, 51, 58, 27, 32, 42, 22, 24, 26, 130, 73, 69, 71, 323, 18, 21, 56, 65, 68, 74, 75, 78, 85, 87, 135, 308, 313, 257, 138, 882, 1011, 35, 46, 62, 70, 29, 30, 77, 34, 72, 37, 317, 338, 60, 875, 49, 61, 393, 38, 39, 126, 324, 339, 19, 36, 40, 43, 23, 122, 80, 127, 133, 136, 28, 66, 104, 101] duplicated
19) lfn 2820000/ff25411f-c560-4080-ae80-d93c31a417a6.root has lumi [5, 10, 12, 14, 25, 27, 45, 60, 28, 30, 42, 313, 424, 134, 333, 38, 284, 286, 325, 330, 67, 68, 73, 120, 425, 921, 972, 128, 16, 26, 2, 4, 7, 21, 31, 56, 17, 63, 51, 133, 32, 47, 82, 9, 23, 1, 3, 33, 44, 20, 66, 50, 8, 13, 35, 39, 6, 24, 29, 19, 48, 135, 11, 15, 304, 307, 127, 65, 72, 53, 897, 126, 85, 86, 18, 22, 34, 740, 119, 55, 49, 166, 40, 41, 64, 324, 129, 36, 46, 132, 305] duplicated
20) lfn 2820000/03c62d39-4b00-4863-9f01-0a6027158258.root has lumi [110, 231, 212, 214, 91, 152, 366, 395, 153, 303, 236, 280, 342, 485, 211, 14, 334, 362, 139, 206, 339, 363, 15, 235, 335, 392, 336, 337, 364, 365, 234, 394, 281, 340, 341, 88, 92, 137, 106, 233, 107, 343] duplicated
21) lfn 2820000/97f86a2e-0d95-4bac-8422-b1e4efc41281.root has lumi [142, 148, 22, 27, 3, 150, 140, 4, 21, 405, 50, 120, 151, 24, 30, 52, 149, 39, 15, 109, 143, 6, 38, 139, 40, 14, 107, 121, 2, 37, 1, 141, 408] duplicated
22) lfn 2820000/08640d0a-9c72-4b64-a945-766b97a95b91.root has lumi [296, 63, 381, 353, 125, 161, 44, 59, 42, 85, 82, 34, 83, 23, 65, 33, 275, 62, 81, 84, 268, 57, 124, 35, 43, 45, 61] duplicated
23) lfn 2820000/550b3a76-e0e0-48c5-8ac2-3e81c25f87ea.root has lumi [218, 514, 321, 178, 454, 231, 179, 546, 512, 229, 513, 552, 511, 554, 553, 455, 155, 526, 488] duplicated
24) lfn 2820000/08531f45-409d-47d8-a99f-4630cbc726fc.root has lumi [151, 95, 216, 246, 377, 243, 301, 32, 743, 86, 74, 97, 98, 152, 83, 163, 814, 104, 111, 162, 310, 354, 31, 242, 78, 198, 300, 381, 75, 123, 297, 153, 106, 108, 288, 370, 376, 96, 290, 371, 414, 30, 73, 84, 109, 161, 103, 196, 380, 110, 112, 289, 291, 815, 107, 87, 241, 105, 355, 287] duplicated
25) lfn 2820000/b7214fa4-7aec-4e50-85e5-557e3e962454.root has lumi [34, 47, 3, 9, 15, 41, 87, 35, 505, 126, 311, 344, 342, 466, 469, 273, 189, 76, 23, 27, 33, 123, 162, 164, 278, 42, 43, 56, 50, 49, 319, 10, 32, 4, 8, 16, 18, 90, 36, 69, 224, 503, 86, 91, 166, 172, 194, 274, 566, 103, 45, 48, 38, 348, 6, 22, 509, 124, 98, 470, 275, 191, 82, 102, 11, 13, 14, 19, 21, 271, 504, 95, 24, 26, 30, 31, 2, 88, 65, 139, 160, 435, 89, 187, 12, 94, 40, 5, 17, 25, 28, 7, 64, 141, 346, 92, 173, 20, 70, 1, 225, 170, 71, 106, 272, 29, 226] duplicated
26) lfn 2820000/f5a95b47-ddbe-45e2-9f8e-2197aaed44f0.root has lumi [172, 173, 141, 167, 170, 171, 153, 152, 111, 169, 168, 112] duplicated
27) lfn 2820000/b96cec87-b22a-44c4-9671-3e939a463bb0.root has lumi [78, 131, 132, 34, 38, 52, 135, 148, 35, 36, 113, 134, 141, 151, 37, 47, 137, 277, 50, 49, 112, 48, 33, 41, 139, 19, 126, 39, 45, 76, 102, 114, 40, 77, 140, 144, 155, 340, 910, 170, 339, 143, 123, 20, 30, 51, 115, 152, 136, 138, 915, 79, 80, 29, 133, 147, 116] duplicated
28) lfn 2820000/f33256fa-baf2-4a11-bd4d-d02291908023.root has lumi [227, 229, 63, 612, 164, 211, 194, 411, 415, 603, 242, 124, 419, 508, 547, 548, 142, 64, 457, 607, 226, 50, 206, 121, 123, 615, 210, 459, 510, 42, 59, 203, 205, 418, 551, 65, 125, 61, 62, 507, 549, 174, 337, 339, 370, 417, 550, 455] duplicated
29) lfn 2820000/cf4201de-4de0-4bda-b3aa-bdd7f26d68a3.root has lumi [40, 379, 436, 573, 574, 1, 3, 11, 101, 116, 138, 80, 78, 202, 172, 189, 9, 187, 419, 114, 183, 113, 118, 210, 445, 446, 420, 131, 401, 231, 248, 284, 199, 447, 472, 177, 178, 10, 17, 117, 108, 130, 193, 421, 500, 422, 277, 499, 484, 229, 46, 12, 70, 18, 228, 279, 132, 2, 41, 280, 323, 169, 45, 418, 79, 16, 103, 324, 5] duplicated
30) lfn 2820000/c390bd15-3d09-465c-a44e-977439a0621c.root has lumi [116, 122, 569, 168, 5, 482, 8, 162, 163, 164, 125, 161, 1, 566, 567, 3, 753, 126, 7, 159, 160, 165, 481, 483, 127, 480, 166, 170, 635, 99, 568, 22] duplicated
31) lfn 2820000/742cbe91-5d1d-410b-b6ee-adaf923ff050.root has lumi [72, 181, 188, 325, 94, 178, 157, 603, 583, 547, 529, 383, 384, 45, 582, 58, 80, 179, 257, 385, 13, 16, 18, 156, 660, 177, 537, 32, 259, 538, 61, 68, 180, 382, 136, 272, 44, 33, 14, 30, 324, 255, 67, 24, 74, 46, 41, 15, 352, 100, 75, 155, 586, 591, 275, 137, 57] duplicated
32) lfn 2820000/f503ce3d-3adb-4a85-8f89-3037d17584ca.root has lumi [458, 187, 5, 481, 8, 16, 191, 19, 201, 202, 9, 257, 649, 315, 179, 259, 327, 330, 7, 200, 261, 482, 328, 223, 68, 80, 192, 6, 72, 73, 180, 471, 483, 4, 18, 17, 251, 178, 224, 74, 79, 15, 260, 262] duplicated
33) lfn 2820000/2c31607b-9982-429c-9e4a-33940e9f11cf.root has lumi [61, 9, 10, 11, 84, 86, 71, 141, 64, 52, 55, 74, 401, 351, 150, 483, 522, 392, 195, 220, 514, 153, 171, 70, 12, 7, 17, 80, 203, 222, 88, 94, 49, 51, 139, 144, 523, 90, 151, 174, 186, 315, 477, 14, 36, 50, 296, 478, 22, 69, 152, 63, 19, 20, 363, 490, 53, 68, 76, 137, 480, 155, 163, 249, 414, 48, 18, 136, 142, 73, 30, 72, 95, 54, 77, 185, 92, 291, 78, 60, 65, 79, 225, 93, 162, 290, 67, 8, 197, 578, 75, 187, 56] duplicated
34) lfn 2820000/85740ac6-b3fa-4409-b946-782855b72586.root has lumi [1139, 1146, 570, 104, 391, 1151] duplicated
35) lfn 2820000/d889e70d-5da0-48c1-9d6f-406d8375d4b8.root has lumi [571, 573, 144, 25, 369, 372, 175, 196, 183, 696, 394, 379, 146, 26, 368, 6, 147, 247, 252, 69, 190, 273, 59, 50, 28, 73, 193, 70, 24, 176, 4, 367, 68, 7, 277, 318, 320, 321, 572, 186, 371, 370, 3, 179, 27, 71, 322, 197, 29, 319, 145, 211, 173] duplicated
36) lfn 2820000/183a01c5-3aff-4033-a33d-468d67fbc857.root has lumi [33, 50, 7, 90, 51, 102, 150, 95, 186, 487, 99, 123, 52, 56, 55, 57, 436, 94, 104, 73, 120, 122, 678, 355, 48, 53, 66, 132, 519, 682, 119, 684, 6, 93, 134, 91, 187, 680, 46, 116, 118, 518, 185, 49, 54, 47, 115, 133, 103, 101, 27, 71, 62] duplicated
37) lfn 2820000/21c16fbe-eab1-4222-a21d-8605c1404f3b.root has lumi [1, 200, 202, 474, 561, 1038, 484, 173, 627, 433, 453, 476, 479, 171, 189, 168, 167, 75, 501, 644, 503, 193, 165, 502, 628, 689, 473, 478, 508, 45, 626, 7, 454, 191, 77, 472, 565, 485, 562, 623, 630, 118, 5, 521, 39, 477, 115, 306, 434, 436, 308, 197, 811, 6, 810, 40] duplicated
38) lfn 2820000/3fe13c92-99cf-406c-9a72-e18e73150992.root has lumi [80, 44, 46, 76, 78, 225, 153, 67, 59, 243, 60, 61, 286, 332, 58, 371, 3, 166, 333, 465, 42, 466, 1, 77, 79, 287, 65, 68, 326, 148, 69, 288, 5, 238, 81, 2, 247, 4, 297, 115, 246, 335, 464, 495, 167, 149, 71, 72, 295, 296] duplicated
39) lfn 2820000/67caa793-d2b6-46f2-98d9-a7691518b263.root has lumi [25, 26, 34, 64, 442, 444, 337, 334, 67, 70, 29, 441, 208, 357, 359, 65, 274, 68, 69, 9, 358, 8, 332, 71, 30, 72, 66, 331, 10] duplicated
40) lfn 2820000/567ba098-9f66-459d-955c-2220caac2fb6.root has lumi [3, 8, 12, 54, 216, 467, 86, 222, 283, 481, 526, 569, 22, 186, 115, 368, 194, 288, 489, 185, 158, 233, 82, 84, 248, 492, 539, 6, 527, 366, 221, 2, 213, 19, 160, 220, 25, 114, 195, 215, 525, 107, 524, 184, 14, 20, 284, 98, 117, 277, 7, 97, 278, 23, 196, 1, 218, 219, 51, 110, 365, 171, 53, 217, 556, 282, 281, 251] duplicated
41) lfn 2820000/f7c90d1e-721f-4e67-8e15-757ebf4390c9.root has lumi [74, 164, 156, 160, 1071, 1068, 136, 132, 133, 131, 158, 1062, 157, 130, 1065, 60, 98, 55, 137, 101] duplicated
42) lfn 2820000/0f7cf88f-6ebe-4cf7-85c0-511f847f60e7.root has lumi [694, 267, 388, 176, 256, 40, 862, 197, 269, 121, 232, 391, 41, 42, 390, 257, 387, 389, 872, 386, 43, 270, 268, 44, 804, 385, 873, 231, 180, 770, 772] duplicated
43) lfn 2820000/62b66ca7-8560-4645-a334-45c32895de65.root has lumi [173, 154, 631, 145, 146, 206, 129, 309, 311, 351, 210, 372, 224, 489, 488, 627, 633, 163, 373, 375, 590, 310, 16, 147, 490, 149, 589, 350, 208, 207, 487, 630, 151, 374, 174, 125, 629, 158, 494, 353, 126, 157, 352, 223] duplicated
44) lfn 2820000/75314751-d8b5-4734-a00c-0961c55fe257.root has lumi [145, 151, 162, 104, 251, 92, 842, 93, 109, 7, 23, 403, 411, 31, 431, 87, 144, 159, 255, 96, 110, 112, 149, 150, 3, 17, 20, 394, 875, 106, 4, 174, 427, 148, 157, 183, 189, 243, 429, 138, 51, 52, 58, 94, 194, 2, 89, 164, 165, 438, 197, 244, 126, 107, 250, 57, 65, 95, 247, 767, 8, 14, 98, 253, 15, 35, 186, 129, 100, 108, 203, 687, 16, 37, 1, 25, 163, 123, 766, 166, 167, 147, 18, 158] duplicated
45) lfn 2820000/757725d9-8a37-4dc5-872b-bf9bcdc8c422.root has lumi [2, 61, 72, 123, 76, 21, 32, 264, 268, 273, 267, 270, 311, 36, 41, 309, 85, 40, 43, 62, 191, 222, 272, 322, 141, 105, 20, 78, 14, 426, 601, 173, 90, 271, 37, 92, 49, 73, 120, 312, 542, 39, 47, 59, 119, 266, 50, 70, 189, 69, 170, 75, 1, 208, 327, 84, 46, 204, 585, 3, 60, 28, 263, 261, 67, 44, 51, 146, 34, 265, 74, 91, 124, 202, 13, 413, 96, 48, 573, 551, 77, 310, 329] duplicated
46) lfn 2820000/be9129cc-a2b6-456d-95f6-c5c7a471e216.root has lumi [37, 43, 204, 353, 359, 360, 4, 9, 10, 36, 224, 227, 361, 372, 460, 230, 290, 8, 47, 56, 278, 20, 84, 118, 129, 147, 94, 149, 150, 206, 68, 16, 18, 32, 358, 5, 231, 355, 67, 279, 19, 21, 83, 145, 127, 164, 7, 86, 103, 48, 154, 356, 2, 3, 26, 33, 74, 34, 91, 76, 79, 85, 120, 223, 62, 11, 350, 35, 376, 61, 14, 168, 12, 6, 93, 109, 157, 38, 386, 1, 89, 130, 155, 44, 225, 207, 87, 280, 203, 77, 88, 13, 98] duplicated
47) lfn 2820000/a510742d-9300-428c-bf0c-25f229917dbf.root has lumi [601, 131, 40, 130, 668, 667, 673, 157, 36, 39, 161, 595, 596, 901, 153, 137, 138, 636, 38, 152, 663, 675, 594, 666, 581, 160, 749, 751, 665, 158, 159, 37, 132, 750, 116, 748, 674] duplicated
48) lfn 2820000/6894665d-0e02-4717-a4ba-de0b83003818.root has lumi [96, 932, 87, 359, 5, 9, 24, 78, 608, 620, 607, 796, 373, 609, 203, 205, 10, 6, 11, 769, 610, 272, 549, 553, 341, 89, 94, 269, 12, 23, 640, 795, 340, 372, 548, 22, 93, 100, 21, 637, 14, 342, 204, 13, 633, 99, 511, 20] duplicated
49) lfn 2820000/8a59878e-55a9-4051-9c29-d0359a79944c.root has lumi [157, 240, 308, 782, 305, 160, 156, 239, 307, 783, 238, 108, 150] duplicated
50) lfn 2820000/b0b0b405-0499-4376-9136-48971810e9a6.root has lumi [5, 228, 329, 81, 89, 92, 93, 87, 88, 229, 86, 230, 300, 82, 94, 110, 502, 91, 90, 272, 593, 227, 176, 79, 80] duplicated
51) lfn 2820000/d4cc6ebb-1a84-477b-932c-e892b3269484.root has lumi [100, 111, 10, 433, 110, 666, 396, 131, 112, 1041, 397, 113, 634, 88, 432, 1045, 398, 2, 434, 755, 764, 1, 635] duplicated
52) lfn 2820000/9736054f-0a2c-48c8-87fe-f506bd12930a.root has lumi [57, 248, 932, 98, 109, 177, 291, 659, 755, 176, 576, 39, 45, 52, 40, 143, 206, 252, 777, 879, 97, 657, 673, 44, 54, 790, 880, 895, 931, 66, 103, 156, 178, 799, 38, 41, 47, 51, 284, 50, 930, 933, 58, 292, 35, 36, 71, 873, 46, 656, 43, 601, 450, 11, 180, 34, 30, 155, 165, 432, 892, 101, 674, 149, 55, 56, 445, 660, 166, 104, 225, 599, 150, 42, 782, 874, 934, 251, 433, 449, 754, 797, 301, 658, 183, 31, 32, 49, 781] duplicated
53) lfn 2820000/6db955e8-7c07-463c-85de-48026f662209.root has lumi [158, 164, 249, 982, 983, 1019, 1042, 2, 12, 247, 253, 9, 22, 14, 18, 72, 73, 979, 159, 197, 254, 5, 17, 19, 20, 238, 252, 264, 1020, 35, 8, 893, 596, 65, 210, 233, 4, 11, 21, 15, 23, 829, 226, 556, 250, 824, 831, 894, 901, 1018, 33, 34, 1034, 271, 16, 30, 64, 552, 237, 248, 261, 826, 904, 7, 10, 13, 976, 1035, 80, 251, 6, 32, 185, 1, 3, 214, 744, 1021, 902, 598, 242, 1081, 635, 597, 903, 975, 984, 985, 36, 193, 27, 671, 741, 823] duplicated
54) lfn 2820000/d459253e-c8d6-4a46-beeb-322d27f6158f.root has lumi [281, 440, 579, 283, 828, 92, 138, 162, 135, 333, 23, 27, 76, 269, 334, 97, 98, 320, 776, 192, 228, 79, 96, 166, 28, 91, 7, 47, 268, 728, 896, 99, 80, 127, 336, 230, 29, 69, 75, 77, 128, 231, 332, 131, 434, 270, 581, 195, 82, 89, 229, 67, 160, 140, 144, 52, 193, 94, 53, 8, 136, 78, 81, 331, 833, 702, 353] duplicated
55) lfn 2820000/e128739e-fccc-461e-b462-8f5d22d7dd8c.root has lumi [15, 16, 37, 81, 83, 100, 154, 170, 174, 182, 199, 212, 215, 228, 256, 25, 28, 50, 53, 55, 59, 62, 75, 77, 89, 96, 132, 140, 286, 705, 731, 900, 210, 217, 732, 41, 120, 893, 814, 258, 259, 279, 623, 818, 3, 18, 24, 33, 46, 56, 58, 84, 99, 112, 141, 158, 183, 207, 11, 12, 14, 23, 32, 34, 52, 54, 60, 61, 87, 94, 98, 609, 708, 196, 197, 202, 209, 683, 35, 905, 164, 277, 5, 10, 22, 30, 57, 68, 74, 76, 88, 126, 156, 214, 262, 290, 9, 44, 63, 71, 92, 95, 133, 789, 155, 198, 206, 717, 223, 1, 686, 168, 349, 350, 819, 72, 27, 79, 80, 85, 124, 169, 191, 194, 226, 263, 20, 40, 43, 45, 47, 51, 78, 136, 684, 200, 205, 218, 29, 201, 203, 235, 70, 211, 2, 4, 8, 21, 65, 69, 93, 97, 105, 138, 157, 162, 179, 213, 227, 6, 13, 17, 19, 31, 49, 66, 86, 285, 153, 189, 270, 257, 352, 39, 82, 91, 135, 163, 36, 42, 48, 67, 894, 309, 261, 73, 7, 121, 26, 38, 64, 288, 681, 107, 351, 90, 125] duplicated
56) lfn 2820000/5f3ac5f3-410e-4d93-ba9d-1ebdc9337dc0.root has lumi [91, 176, 220, 269, 112, 9, 28, 6, 54, 58, 64, 17, 23, 24, 60, 90, 281, 348, 351, 163, 244, 62, 380, 390, 446, 154, 339, 180, 183, 264, 290, 171, 402, 36, 35, 27, 30, 55, 59, 100, 181, 270, 350, 394, 175, 185, 186, 267, 280, 282, 307, 435, 8, 57, 19, 56, 61, 103, 289, 375, 63, 155, 349, 266, 276, 279, 14, 25, 172, 178, 384, 345, 179, 184, 268, 177, 210, 805, 29, 21, 173, 242, 447, 158, 226, 89, 308, 26, 334, 7] duplicated
57) lfn 2820000/35a70d05-fafc-4a25-9d09-b3ee3aa5e228.root has lumi [386, 459, 131, 133, 350, 398, 460, 457, 822, 830, 945, 820, 168, 132, 387, 13, 868, 135, 109, 351, 555, 134, 352, 458, 96, 172, 869, 130, 374, 838, 910] duplicated
58) lfn 2820000/570a4188-54f5-4425-99b1-c86dfef72788.root has lumi [926, 345, 346, 349, 651, 896] duplicated
59) lfn 2820000/c6e61572-6e25-4ee3-9d7c-e349ec5ec5fb.root has lumi [613, 720, 134, 46, 148, 49, 67, 205, 345, 135, 92, 207, 614, 204, 45, 423, 407, 474, 94, 346, 72, 139, 143, 208, 261, 51, 116, 413, 227, 95, 90, 211, 264, 48, 93, 206, 408, 476, 96, 106, 344, 259, 266, 471, 133, 137, 309, 322, 297, 298, 141, 117, 347, 363, 475, 477, 203, 260, 130, 410, 118] duplicated
60) lfn 2820000/637641a0-147f-4c6b-bb46-ddbc1e062371.root has lumi [115, 971, 1033, 2, 973, 475, 21, 23, 30, 32, 5, 24, 28, 29, 11, 19, 27, 39, 66, 86, 87, 88, 41, 55, 13, 119, 327, 77, 1, 12, 20, 6, 7, 34, 8, 18, 22, 25, 26, 36, 43, 9, 16, 56, 65, 10, 1032, 504, 4, 46, 15, 75, 79, 90, 970, 31, 44, 3, 17, 54, 370, 37, 38, 49, 14, 91, 35, 78, 114, 64, 74, 480, 45, 120, 116, 478, 33, 40, 53, 586, 479, 42, 67, 972] duplicated
61) lfn 2820000/5f5a6718-24d8-43b0-ab71-24b11bdcdc70.root has lumi [188, 248, 373, 254, 65, 178, 79, 240, 125, 256, 175, 245, 260, 246, 699, 157, 797, 674, 159, 61, 116, 187, 943, 253, 398, 155, 184, 396, 224, 46, 179, 399, 156, 225, 198, 177] duplicated
62) lfn 2820000/c98c1128-339d-40ae-92b2-bdfc084d175a.root has lumi [734, 761, 779, 940, 46, 268, 764, 78, 690, 769, 49, 247, 650, 832, 841, 248, 45, 257, 730, 92, 269, 521, 649, 61, 93, 846, 848, 54, 763, 840] duplicated
63) lfn 2820000/f205d123-bb3e-427f-84a8-0bf05d108ffa.root has lumi [99, 122, 161, 221, 688, 557, 682, 404, 73, 83, 121, 123, 125, 217, 194, 139, 143, 190, 334, 98, 181, 198, 815, 149, 640, 228, 333, 357, 639, 681, 166, 85, 164, 192, 330, 518, 814, 102, 348, 421, 267, 343, 344, 84, 144, 146, 203, 207, 230, 219, 245, 332, 165, 150, 153, 188, 193, 393, 76, 126, 218, 301, 191, 289, 182, 196, 526, 307, 687, 300, 302, 331, 558, 86, 326, 61, 272, 163, 215, 211, 101, 200, 100, 345, 335, 816, 285, 299, 229, 141, 55, 429, 145, 227] duplicated
64) lfn 2820000/d6a38972-010b-4d84-910b-7d7a4d902581.root has lumi [267, 364, 613, 651, 120, 417, 263, 388, 401, 615, 3, 221, 227, 255, 256, 11, 399, 400, 612, 611, 405, 489, 685, 5, 387, 346, 406, 262, 296, 304, 319, 352, 616, 710, 6, 69, 870, 261, 349, 473, 345, 869, 350, 1, 211, 265, 614, 363, 9, 88, 260, 264, 354, 4] duplicated
65) lfn 2820000/540ab75d-75f0-493f-bf18-ff68a4a25b02.root has lumi [71, 134, 168, 213, 66, 112, 149, 141, 452, 412, 119, 235, 244, 156, 451, 459, 102, 70, 167, 63, 68, 128, 130, 144, 69, 131, 139, 77, 109, 240, 312, 446, 182, 191, 259, 453, 567, 589, 142, 143, 170, 192, 560, 111, 153, 160, 251, 201, 238, 590, 80, 138, 169, 180, 208, 561, 137, 132, 72, 120, 135, 108, 252, 454, 729, 78, 166, 271, 236, 86, 62, 150, 145, 456, 255, 568, 65] duplicated
66) lfn 2820000/b1fe4239-365c-48ec-9a93-fb14da0aa1cf.root has lumi [29, 466, 49, 282, 281, 718, 315, 250, 312, 467, 279, 251, 284, 189] duplicated
67) lfn 2820000/47b7f732-36b7-4d74-8666-413bca9ee09d.root has lumi [67, 25, 33, 137, 147, 36, 589, 40, 202, 24, 37, 50, 2, 38, 141, 199, 493, 21, 22, 142, 200, 35, 30, 31, 23, 591, 8, 603, 39, 164, 590, 34, 32, 146, 201] duplicated
68) lfn 2820000/7ac6edb2-8499-48fb-987e-e795a7456ba3.root has lumi [648, 139, 114, 143, 144, 545, 133, 246, 128, 245, 152, 154, 190, 192, 195, 214, 216, 250, 251, 257, 138, 203, 224, 219, 275, 315, 607, 126, 95, 140, 153, 129, 125, 620, 258, 181, 182, 252, 297, 301, 141, 205, 220, 96, 122, 118, 116, 191, 228, 22, 130, 132, 243, 249, 115, 123, 145, 147, 583, 134, 230, 247, 212, 24, 124, 313, 136, 148, 546, 312, 233, 23, 137, 579, 314, 217, 248, 567] duplicated
69) lfn 2820000/a22e2323-dec6-4f11-b37a-885924092f69.root has lumi [507, 1009, 123, 125, 564, 636, 717, 45, 48, 27, 36, 121, 206, 214, 129, 233, 243, 28, 823, 127, 834, 116, 213, 229, 232, 633, 635, 708, 804, 57, 59, 47, 51, 268, 269, 442, 967, 241, 819, 538, 805, 807, 67, 471, 634, 706, 239, 242, 458, 31, 33, 58, 63, 132, 204, 982, 71, 130, 238, 443, 513, 225, 671, 718, 128, 222, 470, 703, 60, 32, 34, 118, 188, 963, 236, 493, 68, 219, 122, 1008, 210, 469, 472, 961, 49, 447, 195, 218, 434, 705, 965, 808, 25, 672, 547, 983, 52, 98, 960, 707] duplicated
70) lfn 2820000/5a518ae7-2aca-4b40-a3c8-9f0a72a294ad.root has lumi [409, 293, 552, 941, 186, 187, 151, 416, 243, 942, 369, 380, 415, 16, 273, 556, 499, 531, 350] duplicated
71) lfn 2820000/4feed99f-0afe-4697-a72e-be210d931404.root has lumi [12, 60, 79, 337, 455, 497, 57, 569, 155, 190, 345, 444, 477, 80, 49, 51, 54, 59, 93, 98, 144, 247, 16, 11, 134, 181, 189, 333, 461, 462, 9, 125, 87, 156, 96, 103, 105, 151, 341, 419, 479, 498, 503, 510, 101, 13, 82, 89, 92, 137, 496, 52, 58, 319, 480, 131, 47, 90, 109, 140, 141, 146, 238, 6, 7, 5, 62, 183, 339, 17, 160, 94, 99, 230, 340, 415, 537, 467, 66, 417, 875, 53, 67, 117, 124, 332, 509, 132, 110, 145, 129, 186, 187, 416, 306, 317, 318, 91, 464, 180, 88, 10, 133, 139, 95, 128, 342, 418, 666, 14, 460, 315, 316, 81, 48, 97, 108, 142, 143, 169, 157, 8, 18, 159, 489, 512, 136, 463, 50, 170, 158, 430, 492, 15, 130, 478, 100, 154, 185, 414] duplicated
72) lfn 2820000/b14f68cb-54d9-4c68-810e-e439ecb521dd.root has lumi [11, 23, 46, 124, 65, 107, 174, 184, 526, 63, 42, 47, 3, 6, 30, 15, 102, 172, 621, 21, 28, 44, 48, 127, 323, 234, 40, 7, 8, 22, 43, 97, 98, 523, 61, 62, 12, 14, 17, 19, 26, 37, 86, 93, 619, 24, 2, 4, 27, 175, 437, 1141, 5, 39, 59, 626, 64, 106, 528, 112, 13, 531, 543, 18, 173, 318, 368, 442, 622, 41, 1, 9, 38, 440, 67, 179, 246, 525, 176, 20, 89, 667, 236, 315, 394, 563, 443, 70, 103, 60, 183, 393, 624, 673, 16, 178, 92, 45, 935, 10, 530, 566, 25, 85, 29] duplicated
73) lfn 2820000/767edf88-c2da-4042-a0f8-25402a4ea9cc.root has lumi [227, 72, 645, 111, 201, 100, 93, 428, 609, 420, 425, 102, 21, 160, 325, 328, 525, 594, 642, 651, 295, 105, 107, 454, 502, 622, 631, 70, 110, 112, 208, 106, 81, 33, 431, 527, 97, 530, 11, 16, 22, 32, 453, 532, 535, 109, 606, 648, 10, 108, 104, 29, 13, 77, 199, 424, 426, 427, 490, 647, 123, 124, 529, 593, 555, 650, 71, 113, 117, 221, 654, 84, 25, 115, 480, 536, 225, 557, 595, 596, 207, 85, 83, 621, 82, 114, 12, 488, 553, 524, 116, 161, 19, 103, 162, 204, 15, 76, 327, 121, 608] duplicated
74) lfn 2820000/ec6110ac-befb-4ac1-8502-43d2db7559dd.root has lumi [680, 330, 861, 681, 683, 862, 799, 359, 360, 842, 851, 20, 841, 329, 860, 950, 376, 857, 844, 790, 28, 957, 845, 839, 813] duplicated
75) lfn 2820000/4427257a-beab-4fbe-9ce8-65c244f4e7ec.root has lumi [348, 637, 346, 641, 323, 444, 462, 632, 635, 483, 347, 633, 630] duplicated
76) lfn 2820000/e588da22-e690-4a87-8ba7-75830b466b1d.root has lumi [33, 1, 5, 8, 13, 3, 4, 6, 10, 7, 22, 31, 115, 29, 27, 101, 220, 60, 14, 15, 9, 85, 142, 102, 23, 26, 444, 32, 96, 303, 105, 361, 316, 28, 53, 2, 12, 61, 81, 301, 11, 34, 443] duplicated
77) lfn 2820000/73651f63-be07-4aa6-936d-c62f4d49115b.root has lumi [30, 159, 57, 58, 65, 111, 112, 114, 118, 125, 306, 90, 1, 3, 15, 20, 22, 29, 37, 68, 108, 74, 76, 80, 142, 6, 8, 9, 17, 2, 5, 18, 21, 46, 49, 69, 14, 13, 26, 53, 70, 24, 25, 27, 136, 139, 11, 56, 143, 7, 12, 31, 107, 100, 94, 371, 19, 16, 28, 63, 64, 75, 120, 59, 62, 106, 117, 4, 48, 110, 73, 92, 32, 61, 39, 45, 51, 146, 122, 131, 216, 23, 33, 109, 88, 93, 40, 44, 43, 79, 119, 89, 34, 38, 99, 72, 66, 52, 36, 134, 60, 121, 132, 10, 71, 77, 123, 145, 35, 141] duplicated
78) lfn 2820000/9d31e120-d9a9-44de-9674-72400b73c136.root has lumi [105, 152, 23, 85, 121, 578, 8, 11, 119, 120, 21, 36, 526, 13, 175, 84, 193, 350, 437, 25, 49, 512, 513, 570, 10, 581, 712, 760, 17, 281, 430, 569, 585, 86, 22, 83, 75, 248, 52, 87, 47, 48, 140, 81, 72, 510, 499, 500, 507, 523, 50, 300, 572, 39, 51, 20, 80, 82, 714, 7, 24, 176, 344, 14, 200, 128, 53, 709, 214, 64, 224] duplicated
79) lfn 2820000/08b372c5-3dd0-4c53-870d-24b5b687e03a.root has lumi [70, 276, 113, 136, 62, 72, 275, 284, 1156, 44, 108, 32, 63, 226, 1189, 19, 551, 1155, 38, 66, 71] duplicated
80) lfn 2820000/e424c2f9-f0fb-4dff-8290-2fb484d3e5c4.root has lumi [466, 93, 337, 313, 465, 475, 339, 314, 189, 388, 338, 190, 385, 387] duplicated
81) lfn 2820000/5624f968-426d-48c5-ac2d-1528917ead01.root has lumi [145, 578, 226, 415, 976, 978, 575, 576, 414, 472, 234, 126, 163, 268, 980, 224, 135, 574, 225, 128] duplicated
82) lfn 2820000/1f599a68-c6be-4407-9eba-77f4fe8362f9.root has lumi [24, 40, 83, 55, 65, 104, 259, 30, 110, 194, 206, 250, 58, 142, 56, 67, 64, 289, 28, 63, 178, 290, 27, 37, 49, 125, 29, 108, 68, 80, 35, 41, 52, 66, 71, 102, 113, 101, 51, 112, 117, 57, 105, 111, 26, 118, 291, 87, 115, 270, 54, 70, 6, 121, 42, 114, 218, 217, 39, 9, 177] duplicated
83) lfn 2820000/6dea34d5-3b5c-4253-839b-d32d1ada2042.root has lumi [56, 59, 77, 471, 457, 462, 51, 174, 315, 379, 58, 78, 175, 217, 440, 444, 57, 45, 178, 378, 184, 441, 463, 439, 177, 442, 455, 456, 9, 52, 176, 377] duplicated
84) lfn 2820000/feb523a1-13bc-473c-8bf6-460c4b5e64a9.root has lumi [297, 343, 196, 490, 703, 198, 57, 289, 58, 460, 461, 175, 59, 648] duplicated
85) lfn 2820000/1c8ca45c-f537-403c-b3e8-23fe8fe9f858.root has lumi [924, 182, 929, 220, 222, 179, 181] duplicated
86) lfn 2820000/ddf8587e-c33b-4fd2-a82a-0f258ce60761.root has lumi [536, 640, 904, 1059, 1091, 1173, 257, 326, 924, 515, 212, 224, 214, 176, 1189, 946, 1089, 277, 294, 304, 1037, 1046, 213, 16, 1007, 291, 279, 290, 293, 298, 299, 190, 325, 356, 399, 951, 320, 922, 925, 930, 597, 957, 958, 1023, 1036, 1044, 1157, 305, 295, 322, 1170, 191, 595, 210, 273, 359, 950, 328, 599, 265, 953, 1142, 232, 296, 410] duplicated
87) lfn 2820000/d30d3bbd-1e0d-40f4-8f7c-8010677c287e.root has lumi [114, 119, 2, 13, 18, 23, 32, 33, 40, 44, 55, 58, 59, 47, 49, 145, 1387, 1, 3, 9, 34, 37, 39, 43, 50, 53, 89, 90, 25, 26, 56, 60, 110, 126, 135, 148, 151, 177, 180, 4, 179, 8, 14, 16, 19, 22, 24, 41, 117, 1170, 1376, 1398, 1407, 1463, 138, 17, 28, 52, 64, 150, 1081, 1158, 1173, 1429, 1432, 1437, 1459, 1462, 10, 36, 38, 147, 5, 45, 46, 57, 122, 124, 125, 137, 155, 156, 161, 118, 30, 31, 116, 1176, 1468, 1375, 1382, 152, 48, 818, 989, 1112, 20, 27, 21, 29, 63, 86, 88, 109, 111, 123, 128, 7, 15, 113, 178, 12, 127, 1460, 1469, 1381, 1431, 1436, 120, 6, 42, 54, 146, 1108, 11, 87, 129, 176, 1465, 815, 35, 51, 61, 105, 136, 115, 121, 1172, 1174, 1458, 153, 62, 1427, 1461, 112, 1344, 1361, 201, 1464] duplicated
88) lfn 2820000/422ddb30-78c7-4c80-b6d5-c4229fa0c850.root has lumi [543, 570, 971, 545, 801, 539, 540, 544, 607, 278, 63, 563, 1073, 1309, 546, 611, 222, 485, 55, 56, 58, 57, 534, 559, 568, 1209, 996, 997, 1072, 1200, 365, 592, 606, 612, 532, 474, 59, 548, 608, 537, 609, 259, 375, 556, 360, 51, 52, 60, 61, 605, 366, 451, 49, 53, 50, 538, 557, 531, 54, 62, 528, 333] duplicated
89) lfn 2820000/947ac75b-d954-443d-81fe-4785f05885cf.root has lumi [1032, 1252, 725, 36, 23, 27, 40, 720, 32, 33, 1077, 1059, 1124, 1201, 1374, 1400, 1005, 1080, 34, 237, 1061, 1069, 1113, 1140, 1238, 1246, 1270, 1271, 218, 1025, 1082, 1152, 26, 22, 184, 1068, 1273, 1409, 646, 1269, 587, 1063, 743, 1071, 1237, 1274, 1373, 1031, 41, 24, 30, 1321, 1074, 31, 1066, 1070, 1168, 1067, 1088, 1275, 25] duplicated
90) lfn 2820000/4b6646ca-cd45-4cb1-8652-51dfab4887b4.root has lumi [220, 221, 219] duplicated
91) lfn 2820000/c899ad1d-3a20-4411-9a7a-8d175eb33b99.root has lumi [1086, 1131, 186, 590, 724, 1087, 1402, 1457, 260, 618, 928, 1075, 584, 882, 896, 192, 973, 694, 883, 969, 972, 1080, 1094, 586, 1132, 965, 886, 970, 976, 1104, 188, 1021, 1130, 1456, 836, 971, 1044, 727, 885, 1057, 1089, 599, 585, 1040, 587, 673, 888, 1403, 583, 243, 1095, 1416, 722, 1019] duplicated
92) lfn 2820000/16a3a6ab-8b26-4093-a241-51372b482c48.root has lumi [303, 318, 499, 287, 469, 266, 267, 289, 420, 286, 292, 316, 424, 262, 270, 284, 319, 470, 317] duplicated
93) lfn 2820000/9b704476-7fc1-44d4-bfa6-6f53d6f1daab.root has lumi [70, 76, 69, 927, 1042, 1054, 1110, 1191, 1220, 1334, 1335, 779, 1011, 1020, 193, 253, 1055, 785, 1012, 1048, 1049, 1168, 1255, 68, 75, 77, 59, 71, 108, 928, 1128, 1222, 100, 117, 119, 170, 1079, 1135, 778, 970, 1047, 1064, 252, 67, 79, 116, 65, 72, 240, 241, 652, 1115, 1122, 735, 1052, 740, 929, 931, 1256, 66, 113, 115, 81, 103, 104, 926, 1019, 1124, 1223, 862, 1092, 1096, 1125, 1127, 114, 972, 1014, 736, 1013, 1046, 111, 112, 78, 1017, 1053, 786, 245, 1149, 1147, 1221, 154, 99, 118, 121, 98, 152, 741, 653, 1038, 109] duplicated
94) lfn 2820000/772ee8ea-8df5-4b41-95de-901b20514b02.root has lumi [418, 575, 1192, 1310, 1313, 1430, 1009, 1113, 1404, 95, 88, 165, 1199, 85, 160, 167, 168, 982, 988, 1114, 1144, 1148, 1290, 1406, 983, 1193, 86, 96, 121, 84, 107, 161, 162, 900, 1135, 145, 146, 166, 578, 580, 980, 120, 150, 151, 1281, 1303, 80, 1250, 48, 91, 94, 101, 147, 313, 105, 629, 1162, 93, 171, 985, 1118, 1467, 143, 149, 1357, 92, 202, 82, 103, 158, 824, 1156, 1291, 1016, 122, 182, 1155, 83, 142, 157, 624, 90, 1247, 148, 106, 579, 1182, 89, 47, 81, 104, 581, 1181, 1358, 1205] duplicated
95) lfn 2820000/dea14f99-b585-49ca-9305-3a4536e578da.root has lumi [186, 191, 138, 194, 773, 276, 270, 1159, 1229, 170, 246, 188, 139, 190, 197, 1120, 1453, 174, 275, 1122, 1157, 177, 279, 999, 268, 1045, 1294, 675, 1218, 1234, 247, 283, 1004, 1121, 1296, 1304, 995, 140, 196, 1454, 1153, 165, 990, 1152, 1376, 269, 271, 1119, 228, 248, 1227, 168, 175, 1151, 274, 1377, 195] duplicated
96) lfn 2820000/92699f01-bce3-4895-857a-32189e3a06e3.root has lumi [210, 347, 522, 127, 135, 206, 1426, 132, 209, 330, 331, 449, 229, 1338, 128, 1333, 557, 1474, 130, 131, 1352, 521, 134, 329, 133, 246, 343, 1349, 208, 1331] duplicated
97) lfn 2820000/4b764b12-8d0f-493b-94ea-eb1e2bd09600.root has lumi [314, 318, 320, 256, 269, 199, 315, 229, 614, 306, 321, 202, 253, 303, 456, 458, 452, 159, 230, 260, 299, 304, 319, 198, 455, 200, 305, 317, 450, 233, 254, 316, 457, 251, 232, 231] duplicated
98) lfn 2820000/4491eadb-0ae7-4d0b-adba-6aa246d7e9a7.root has lumi [434, 438, 262, 264, 225, 802, 241, 245, 273, 279, 224, 236, 283, 222, 277, 666, 244, 238, 335, 265, 267, 284, 271, 280, 414, 555, 660, 667, 332, 258, 281, 243, 261, 278, 372, 657, 250, 240, 668, 374, 560, 266, 282, 369, 242, 269, 239] duplicated
99) lfn 2820000/9706e1ce-a4b0-4c60-992b-668fc415ffb0.root has lumi [235, 98, 79, 485, 63, 540, 62] duplicated
100) lfn 2820000/d6c0e578-c55b-4da7-b763-dfdc2e1024fb.root has lumi [51, 156, 408, 857, 66, 69, 73, 38, 57, 70, 75, 99, 1058, 20, 39, 45, 46, 52, 60, 856, 72, 21, 67, 1253, 720, 54, 493, 37, 860, 1003, 19, 78, 100, 669, 736, 859, 96, 189, 864, 71, 1055, 1252, 492, 40, 1002, 74, 44, 1054, 412, 409, 58, 25, 706, 1053, 738, 97, 42, 59, 858] duplicated
101) lfn 2820000/5ad672c4-d672-4450-8bc4-c8cbbdc28c9c.root has lumi [556, 357, 1110, 84, 394, 355, 83, 362, 88, 354, 369, 85] duplicated
Total Duplicated Lumis MinAOD : 372 unique lumis that are duplicated -> 865 sum of duplicated lumis Total Duplicated Lumis NanoAOD : 5138 unique lumis that are duplicated -> 11849 sum of duplicated lumis
@amaltaro can you tell me how to verify the duplicates from root? I have downloaded one file.
@hassan11196 Ahmed, I have not yet checked the list you provided above. But as we quickly discussed today about the duplicate lumis inside the same root file, I would suggest you reaching out to Qier and Luca to see how you can open a root file with root and inspect its content. It's been a while that I have done that and I no longer remember.
Hi @amaltaro,
I was reviewing one of the files[1] mentioned above using the dbs API and noticed that they had duplicate lumis but different run numbers.
It seems that the getFileLumis function in the notebook is checking only lumis and not lumi run pair. I modified the code to check for lumi-run pairs and found no duplicates in both output datasets.
Before:
def getFileLumis ...
....
if lumi in resp[lfn]:
After:
def getFileLumis ...
...
if (lumi,run_number) in resp[lfn]:
So can you confirm that this was a false alarm? The new question I have is why does this workflow have more lumis in NanoAOD then in MiniAOD?
[1] /store/data/Run2023C/Muon1/MINIAOD/22Sep2023_v4-v2/2820001/99d5cc32-2fa7-4713-884f-1d3361e848d3.root
@hassan11196 Ahmed, if those same lumis belong to different run numbers, then it is definitely NOT a duplication. Thank you very much for spotting that. I will take this opportunity and update the python notebook in my repository (but if you prefer, feel free to share your current code and I can push that in as well).
Maybe we should re-do such tests with the previous 22 (?) workflows that you reported with dup lumis?
Hi @amaltaro, For the above 22 workflows the duplicate lumis are across files and code used takes in accounts run number and lumi key[1]. So they should be real duplicates, as I precaution I will still review it again.
[1] https://gitlab.cern.ch/CMSProductionReprocessing/WmAgentScripts/-/blob/python3-migration-docker-setup/src/python/Services/DBS/DBSReader.py#L499
Thank you
Hi @amaltaro @hassan11196 @haozturk , I confirm what is reported in the last message from Ahmed. I think we can proceed with understanding the source of the duplicated lumis across different files
I am afraid that these workflows are now old enough and we are no longer able to find their logs anywhere.
In addition, it looks like we are not collecting logs for production/processing tasks in StepChain, because I could only find logs for merge jobs under CERN EOS:
/eos/cms/store/logs/prod/2024/02/WMAgent/cmsunified_task_EXO-Run3Summer22MiniAODv4-00662__v1_T_240213_134751_3377
and these 2 tarballs contains Merge logs for the NANOAODSIM output with duplicate lumi number 7:
cmsunified_task_EXO-Run3Summer22MiniAODv4-00662__v1_T_240213_134751_3377-EXO-Run3Summer22NanoAODv12-00662_0NANOEDMAODSIMoutputMergeLogCollect-c01-012-162-10-logs.tar
cmsunified_task_EXO-Run3Summer22MiniAODv4-00662__v1_T_240213_134751_3377-EXO-Run3Summer22NanoAODv12-00662_0NANOEDMAODSIMoutputMergeLogCollect-heplnc128-1-logs.tar
--> WMTaskSpace_Lumi7_2/logCollect1/9efb296c-78fb-4dd8-8756-607120d30dda-3-0-logArchive.tar.gz
These are contents from the FJR from the relevant merge jobs:
<InputFile>
<LFN>/store/unmerged/Run3Summer22NanoAODv12/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/NANOAODSIM/130X_mcRun3_2022_realistic_v5-v4/40000/e64bd0bd-0b20-436e-9a22-d99b3a5453c4.root</LFN>
<Run ID="1">
<LumiSection ID="7"/>
</Run>
<File>
<LFN>/store/mc/Run3Summer22NanoAODv12/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/NANOAODSIM/130X_mcRun3_2022_realistic_v5-v4/40000/Merged.root</LFN>
<GUID>f514eb6c-47ce-4520-9d23-7f3523365fa9</GUID>
and
<InputFile>
<LFN>/store/unmerged/Run3Summer22NanoAODv12/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/NANOAODSIM/130X_mcRun3_2022_realistic_v5-v4/40000/08aa3acb-baf5-4cc7-91b8-524ffd40c100.root</LFN>
<Run ID="1">
<LumiSection ID="7"/>
</Run>
<File>
<LFN>/store/mc/Run3Summer22NanoAODv12/ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/NANOAODSIM/130X_mcRun3_2022_realistic_v5-v4/40000/Merged.root</LFN>
<GUID>9b81675a-8a98-476f-8482-8a061d054adb</GUID>
My suggestion is then to either invalidate things and move forward, or start these workflows from scratch. @hassan11196 @hassan11196
This is a small workflow though cmsunified_task_EXO-Run3Summer22MiniAODv4-00662__v1_T_240213_134751_3377 and it would be interesting to clone it to see if the problem reoccurs. Hassan, Ahmed, is it something that you could do?
From the google document, I see that the last workflow has been reported with duplicate lumis: cmsunified_task_EXO-Run3Summer22MiniAODv4-00662__v1_T_240213_134751_3377
I would suggest you to follow up on that one (either invalidate the relevant files; or produce that dataset all over again). Input data is /ZPrime2DarkPhoton_HMass-1000_DPMass-0p3_TuneCP5_13p6TeV-pythia8/Run3Summer22DRPremix-124X_mcRun3_2022_realistic_v12-v2/AODSIM
Hi @amaltaro Here are a few recent workflows with duplciated lumis https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_EXO-RunIISummer20UL18wmLHEGEN-05236 https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_EXO-RunIISummer20UL18wmLHEGEN-05315 https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_EXO-RunIISummer20UL18wmLHEGEN-05375
@hassan11196 Ahmed, can you please also share your modified version of the initial jupyter notebook?
Thanks Ahmed! I finally managed to look into one of the workflows that we cloned from production to testbed:
amaltaro_SC_PU_5Steps_June2024_Val_240627_215847_1521
and there are no duplicate lumis in the output data, as can be seen below:
Summary for: None
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 0 | 0 | 0 |
| num_file | 0 | 0 | 0 |
| num_event | 0 | 0 | 0 |
| num_lumi | 0 | 0 | 0 |
Summary for: /RPVStopStopToJets_UDD323_M-2000_TuneCP5_13p6TeV-madgraphMLM-pythia8/Run3Summer22EEDRPremix-EXO_Run3Summer22EEDRPremix_01310_1_SC_PU_5Steps_June2024_Val_Alanv3-v11/AODSIM
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 1 | 1 | 1 |
| num_file | 240 | 240 | 240 |
| num_event | 1514571 | 1514571 | 1514571 |
| num_lumi | 2405 | 2405 | 2405 |
Summary for: /RPVStopStopToJets_UDD323_M-2000_TuneCP5_13p6TeV-madgraphMLM-pythia8/Run3Summer22EEMiniAODv4-EXO_Run3Summer22EEMiniAODv4_01191_0_SC_PU_5Steps_June2024_Val_Alanv3-v11/MINIAODSIM
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 1 | 1 | 1 |
| num_file | 72 | 72 | 72 |
| num_event | 1514571 | 1514571 | 1514571 |
| num_lumi | 2405 | 2405 | 2405 |
Summary for: /RPVStopStopToJets_UDD323_M-2000_TuneCP5_13p6TeV-madgraphMLM-pythia8/Run3Summer22EENanoAODv12-EXO_Run3Summer22EENanoAODv12_01191_0_SC_PU_5Steps_June2024_Val_Alanv3-v11/NANOAODSIM
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 1 | 1 | 1 |
| num_file | 4 | 4 | 4 |
| num_event | 537513 | 537513 | 0 |
| num_lumi | 854 | 854 | 854 |
Based on that, I am inclined to say that there is nothing particular to the workflow and/or job splitting that triggers this duplicate run/lumi. The test workflow above has the same job splitting as the original one (and no input dataset): https://cmsweb.cern.ch/reqmgr2/fetch?rid=cmsunified_task_EXO-Run3Summer22EEwmLHEGS-01479__v1_T_240206_120800_9709
so I don't think it would be an edge case performing the job splitting. More debugging is needed though.
@amaltaro @hassan11196 maybe we can consider running a backfill workflow with the same config, to confirm this result? IMU, backfill agents are closer to the production agents
Andrea provided me a list of duplicated AODSIM run/lumis for the following dataset: for cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728
Zooming in into the following files with duplicate run/lumi:
"1:147": [
"/store/mc/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/B3FA4D01-5709-1743-83D4-52E0BB3E4AD1.root",
"/store/mc/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/C403A9BB-15AB-FE41-A821-2539DC31978A.root"
],
"1:6": [
"/store/mc/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/B3FA4D01-5709-1743-83D4-52E0BB3E4AD1.root",
"/store/mc/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/C403A9BB-15AB-FE41-A821-2539DC31978A.root"
],
we managed to find the merge job log for "1:6" file B3FA4D01-5709-1743-83D4-52E0BB3E4AD1.root in submit7
install/wmagentpy3/JobArchiver/logDir/c/cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728/JobCluster_2581/Job_2581417.tar.bz2
while the second file C403A9BB-15AB-FE41-A821-2539DC31978A.root for run/lumi "1:6" is in vocms0252:
install/wmagentpy3/JobArchiver/logDir/c/cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728/JobCluster_1790/Job_1790429.tar.bz2
@@@@@@@@@@@
Based on these merge job log files, we grab the list of unmerged input files for each of them.
For the first merged file B3FA4D01-5709-1743-83D4-52E0BB3E4AD1.root, here is the list of unmerged files:
Set attribute process.source.fileNames to cms.untracked.vstring(
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/9C150C2A-D0EC-2648-B739-D67ED221B37B.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/C789420C-FBA5-DD49-A1FB-02D35B342CE7.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/80C317FE-9EDB-2D49-9EBA-6E69CBF5E565.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/313672D9-F962-6141-92BF-7B6D942EA56A.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/BF12AB84-2027-6949-B438-1E96EC890472.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/4E126549-50F4-994F-A369-E323DACBCA82.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/A357EC32-BCAD-924B-A420-7D1427DFE34A.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/70000/F0220A0B-84A4-8144-9A6E-93F4FABF6FC5.root'
)
While for the second merged file C403A9BB-15AB-FE41-A821-2539DC31978A.root, here is the list of unmerged files:
Set attribute process.source.fileNames to cms.untracked.vstring(
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/CD61A727-E6BE-9E49-B48B-7C0A6A78B9AF.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/F3BDB24D-40FD-514D-88FE-8795C6806DC2.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/B7636930-6DE3-D249-BF25-EAC3B517C25E.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/5CCC30E1-BDA1-4144-BBD2-253B47839093.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/E568C908-AEDB-E14B-A0E4-CC530997687C.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/75C1935E-922F-404D-9E7B-51C5E709E0B7.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/E52B9415-ECCA-1E44-B649-3F3DD2D42A73.root',
'/store/unmerged/RunIISummer20UL18RECO/WRtoNTautoTauTauJJ_WR3000_N400_TuneCP5_13TeV_madgraph-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v2/2520000/FD835AB7-DA43-2643-9942-5D124170AFAA.root'
)
@@@@@@@@@@@
Now we have to search for the log files for each of these unmerged files, until we can lock in in the file with "1:6".
For the first merged file, if we search for the unmerged file 9C150C2A-D0EC-2648-B739-D67ED221B37B.root, here is the log file (submit7):
install/wmagentpy3/JobArchiver/logDir/c/cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728/JobCluster_2505/Job_2505155.tar.bz2
and the job description says:
'lfn': 'MCFakeFile-cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728-EXO-RunIISummer20UL18wmLHEGEN-05236_0-e2f4c95a28ce5e64dc79a4bcb88fb3d8', 'FirstEvent': 8756, 'LastEvent': 10506, 'FirstLumi': 6, 'LastLumi': 6, 'FirstRun': 1, 'LastRun': 1...
For the second merged file, if we search for the unmerged file CD61A727-E6BE-9E49-B48B-7C0A6A78B9AF.root, here is the log file (vocms0252):
install/wmagentpy3/JobArchiver/logDir/c/cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728/JobCluster_1719/Job_1719546.tar.bz2
and the job description says:
'lfn': 'MCFakeFile-cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728-EXO-RunIISummer20UL18wmLHEGEN-05236_0-e2f4c95a28ce5e64dc79a4bcb88fb3d8', 'FirstEvent': 8756, 'LastEvent': 10506, 'FirstLumi': 6, 'LastLumi': 6, 'FirstRun': 1, 'LastRun': 1,
For the record, the script used for searching for a given file id name inside the tarballs is: https://raw.githubusercontent.com/amaltaro/ProductionTools/master/untarLogArchive.py
Now if you only want to read about the actual culprit -at least for this workflow - the problem seems to be with WorkQueueManager, which actually acquired the same WQE element in multiple agents!
Log from submit7:
2024-06-06 12:34:14,032:139861761881856:INFO:WorkQueueBackend:Accepting workflow: cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728, with prio: 110000, element id: bfb7032a6cdc0a8186be96eb830df31a, for site: T2_CH_CERN_P5
...
2024-06-06 12:41:15,031:139861761881856:INFO:WorkQueue:Queuing element bfb7032a6cdc0a8186be96eb830df31a for /cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728/EXO-RunIISummer20UL18wmLHEGEN-05236_0 with policy MonteCarlo, with 151 job(s) and 151 lumis on events 1-262776
log from vocms0252:
2024-06-06 19:29:47,873:139992078464768:INFO:WorkQueueBackend:Accepting workflow: cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728, with prio: 110000, element id: bfb7032a6cdc0a8186be96eb830df31a, for site: T2_CH_CERN_P5
...
2024-06-06 19:37:45,975:139992078464768:INFO:WorkQueue:Queuing element bfb7032a6cdc0a8186be96eb830df31a for /cmsunified_task_EXO-RunIISummer20UL18wmLHEGEN-05236__v1_T_240606_163348_8728/EXO-RunIISummer20UL18wmLHEGEN-05236_0 with policy MonteCarlo, with 151 job(s) and 151 lumis on events 1-262776
there is an offset of about 5min between these agents. Actually, even if it was an offset of half second, this should NEVER happened.
This is a serious bug that needs to have high priority attention! I am going to create a new ticket to properly reflect this problem; then we can decide whether we close this or not (or if we close it together with the to-be-created ticket).
UPDATE: here is the ticket https://github.com/dmwm/WMCore/issues/12041
Thank you for narrowing down the issue @amaltaro. Do let me know if I can help in replicating the WQE being picked by multiple agents.
Hi @amaltaro, As I mentioned in today's WMCore Dev meeting, I found a recent workflow acquired after 2 August 2024 to have duplicated lumis in Output.
Here are the details,
workflow name: cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194
Summary for: /JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/Run3Summer22DRPremix-124X_mcRun3_2022_realistic_v12-v3/AODSIM
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 48 | 48 | 48 |
| num_file | 362 | 362 | 362 |
| num_event | 1871312 | 1871312 | 1871312 |
| num_lumi | 21318 | 21318 | 21318 |
Summary for: /JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/Run3Summer22MiniAODv4-130X_mcRun3_2022_realistic_v5-v3/MINIAODSIM
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 50 | 50 | 50 |
| num_file | 126 | 126 | 126 |
| num_event | 1876953 | 1876953 | 1876953 |
| num_lumi | 21380 | 21381 | 21381 |
Summary for: /JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/Run3Summer22NanoAODv13-133X_mcRun3_2022_realistic_ForNanov13_v1-v3/NANOAODSIM
| metric | dataset level | block level | file level |
|-----------|-----------------|---------------|--------------|
| num_block | 47 | 47 | 47 |
| num_file | 91 | 91 | 91 |
| num_event | 1875596 | 1875596 | 0 |
| num_lumi | 21364 | 21365 | 21365 |
Output Dataset ['Run3Summer22DRPremix-124X_mcRun3_2022_realistic_v12-v3', 'AODSIM'] -> produced by 1 wf
Output Dataset ['Run3Summer22MiniAODv4-130X_mcRun3_2022_realistic_v5-v3', 'MINIAODSIM'] -> produced by 1 wf
Output Dataset ['Run3Summer22NanoAODv13-133X_mcRun3_2022_realistic_ForNanov13_v1-v3', 'NANOAODSIM'] -> produced by 1 wf
Map of duplicate files and lumis is:
AODSIM Dataset:
MINIAODSIM Dataset:
Lumi {(6191, 1)} is common between
1. -> /store/mc/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810000/44d02b3e-0919-4e9b-bb83-9e783f76a4de.root
2. -> /store/mc/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810000/382937da-9cc3-4638-925c-36abad87887f.root
NANOAODSIM Dataset:
Lumi {(6191, 1)} is common between
1. -> /store/mc/Run3Summer22NanoAODv13/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/NANOAODSIM/133X_mcRun3_2022_realistic_ForNanov13_v1-v3/2810000/3a8c5378-bb96-4f01-ae50-df0ff50249d7.root
2. -> /store/mc/Run3Summer22NanoAODv13/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/NANOAODSIM/133X_mcRun3_2022_realistic_ForNanov13_v1-v3/2810000/9a780611-2c50-4c2a-8a4c-ece1477ce3d9.root
Do let me know if you need anything else.
Thank you.
I still don't have a final answer on what happened with the workflow above, but I can confirm that it is not the same issue that we were having - multiple agents pulling the same WQE - as I couldn't see anything in the other agents, only vocms0281 worked on this workflow.
I confirmed that the duplicate lumi 6191 is in 2 different job report files, and the unmerged files that produced it are:
"runs": { "1": { "6191": null } },
"lfn": "/store/unmerged/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810016/e270eb71-c267-413b-bf7e-889fc4c36094.root",
"events": 88
and
"events": 89,
"lfn": "/store/unmerged/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810016/2912906c-4046-4014-9fdf-7a5705dbc57c.root",
"runs": { "1": { "6191": null } }
Using a modified version of untarLogArchive.py, here are all of the relevant logs for these 2 files above (and their relevant job type):
xxx@vocms0281:srv $ python3 untarLogArchive.py /data/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194 e270eb71-c267-413b-bf7e-889fc4c36094.root | grep MATCH
MATCH! LogArchive /data/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_4082/Job_4082746.tar.bz2 Merge
MATCH! LogArchive /data/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_4091/Job_4091512.tar.bz2 Cleanup
and
cmst1@vocms0281:srv $ python3 untarLogArchive.py /data/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194 2912906c-4046-4014-9fdf-7a5705dbc57c.root | grep MATCH
MATCH! LogArchive /data/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_3956/Job_3956040.tar.bz2 Production
MATCH! LogArchive /data/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_4131/Job_4131004.tar.bz2 Cleanup
MATCH! LogArchive /data/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_4123/Job_4123669.tar.bz2 Merge
As can be seen, this script does not find any occurrence of e270eb71-c267-413b-bf7e-889fc4c36094.root in a Production job, in the wmagentJob.log file. So I cannot say how exactly that job was configured and why we requested the same lumi in 2 different jobs.
The only explanation I can give for that is that the job producing that file was actually retried somehow, hence overwriting the wmagentJob.log with the content of the new retry. But, if it is retried, it cannot be successful and that file cannot get assigned to a merge job... so this hypothesis is pretty much weak!
I decided to revisit this workflow above and try to make sense of the duplicate lumis. Here are new important findings.
I looked again into the Production job previously mentioned, copying it here:
/data/dockerMount/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_3956/Job_3956040.tar.bz2
and noticed it was retried once.
Given that wmagentJob.log is rewritten upon job retry, I decided to look at the condor logs stored in this tarball. Here are the MINIAODSIM files produced by that job in different retries:
Job_3956040/condor.61071.8.err (retry=0)
LFN: /store/unmerged/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810016/e270eb71-c267-413b-bf7e-889fc4c36094.root
and
Job_3956040/condor.61261.92.err (retry=1)
LFN: /store/unmerged/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810016/2912906c-4046-4014-9fdf-7a5705dbc57c.root
If the job was retried, then it must have been seen as failed by JobAccountant before being accepted as a successful job. This can be confirmed in the component logs:
xxx@vocms0281:current $ grep '3956040' install/JobAccountant/ComponentLog*
install/JobAccountant/ComponentLog.1:2024-08-07 16:11:58,579:140543013328640:WARNING:AccountantWorker:Job 3956040 , bad jobReport, failing job
...
install/JobAccountant/ComponentLog.1:2024-08-08 01:25:27,166:140543013328640:INFO:AccountantWorker:Job 3956040 , handle successful job
Now we can look into the merge jobs that produced these 2 files (as reported by Ahmed above):
MINIAODSIM Dataset:
Lumi {(6191, 1)} is common between
1. -> /store/mc/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810000/44d02b3e-0919-4e9b-bb83-9e783f76a4de.root
2. -> /store/mc/Run3Summer22MiniAODv4/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/MINIAODSIM/130X_mcRun3_2022_realistic_v5-v3/2810000/382937da-9cc3-4638-925c-36abad87887f.root
and those logs are reported in my post above, which I also copy here:
/data/dockerMount/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_4082/Job_4082746.tar.bz2
/data/dockerMount/srv/wmagent/current/install/JobArchiver/logDir/c/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/JobCluster_4123/Job_4123669.tar.bz2
where the merge job Job_4082746 actually processed this file e270eb71-c267-413b-bf7e-889fc4c36094.root (which should have never been merged - as it belongs to a fail job), while the second merge job Job_4123669 processed 2912906c-4046-4014-9fdf-7a5705dbc57c.root.
As shown in the comment above, the output files of a successful job that was marked as failed in JobAccountant (hence, it being an actual failed job) was actually fed as input for merge jobs. This is a bug and it should have never happened!
Looking into JobAccountant log, I see this:
2024-08-07 16:11:58,575:140543013328640:INFO:AccountantWorker:Handling /data/srv/wmagent/current/install/JobCreator/JobCache/cmsunified_task_BPH-Run3Summer22GS-00163__v1_T_240806_141923_3194/BPH-Run3Summer22GS-00163_0/JobCollection_313818_0/job_3956040/Report.0.pkl
2024-08-07 16:11:58,578:140543013328640:WARNING:AccountantWorker:Job 3956040 accepted for multi-step CMSSW, even though the expected outputModules does not match content of the FWJR.
2024-08-07 16:11:58,579:140543013328640:WARNING:AccountantWorker:The following file does not have any location: {'lfn': '/store/unmerged/Run3Summer22DRPremix/JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen/AODSIM/124X_mcRun3_2022_realistic_v12-v3/2810016/087a60ad-655e-4eb8-94a3-6d07d088a9a1.root', 'size': 38562758, 'events': 88, 'checksums': {'adler32': 'f8d1041e', 'cksum': '3468691647'}, 'runs': {<WMCore.DataStructs.Run.Run object at 0x7fd2afeb6670>}, 'merged': False, 'last_event': 0, 'first_event': 0, 'locations': set(), 'parents': set(), 'pfn': '/srv/job/WMTaskSpace/cmsRun3/AODSIMoutput.root', 'branches': [], 'input': [''], 'inputpfns': ['file:../cmsRun2/PREMIXRAWoutput.root'], 'branch_hash': '66aedf5878c8cb3b708d7fca0fa6bce1', 'catalog': '', 'guid': '087a60ad-655e-4eb8-94a3-6d07d088a9a1', 'module_label': 'AODSIMoutput', 'dataset': {'applicationName': 'cmsRun', 'applicationVersion': 'CMSSW_12_4_16', 'primaryDataset': 'JPsiMuMuMuMu_JPsiNoFilter_4MuPtEtaFilter_TuneCP5_13p6TeV-pythia8-evtgen', 'processedDataset': 'Run3Summer22DRPremix-124X_mcRun3_2022_realistic_v12-v3', 'dataTier': 'AODSIM'}, 'acquisitionEra': 'Run3Summer22DRPremix', 'processingVer': 3, 'validStatus': 'PRODUCTION', 'globalTag': '124X_mcRun3_2022_realistic_v12', 'prep_id': 'BPH-Run3Summer22DRPremix-00185', 'configURL': 'https://cmsweb.cern.ch/couchdb;;reqmgr_config_cache;;3c9eba5165582b80e1c9b828b7b0945d', 'inputPath': None, 'outputModule': 'AODSIMoutput', 'fileRef': <WMCore.Configuration.ConfigSection object at 0x7fd2aef2a730>}
2024-08-07 16:11:58,579:140543013328640:WARNING:AccountantWorker:Job 3956040 , bad jobReport, failing job
Based on this log, I am inclined to say that we have a bug on the way JobAccountant deals with these ill cases where an output file is reported as not having any location.
Thanks to Git blame, I can see that this issue was likely caused by a pull request I provided last year: https://github.com/dmwm/WMCore/pull/11433
@hassan11196 given that this duplicate lumi reason is completely different than what we were debugging, I would like to close this issue out an open a new bug ticket to address the problem aforementioned in WMAgent.
@hassan11196 I have created the following issue https://github.com/dmwm/WMCore/issues/12092
to address the recent duplicate lumi issue, with root cause on an ill handling of an exceptional job report in JobAccountant.
I am closing this issue out, but please let me know if there is anything else missing. Thank you for all your help so far!