AliceO2
AliceO2 copied to clipboard
Published
20 hours ago •
AliceO2Group
AliceO2Group
TRD raw reader update
- Tracklet parsing moved into CruRawReader
- prepared Link ID to HCID mapping in CCDB
Still to be done are digit parsing and the event record stuff, this will follow after this is through. Then one can also activate the new link mapping. Also a few error messages might need to be adapted, but this can also be done with the next PR
A few more words, since this turned out to become a rewrite of most of the actual parsing of the TRD raw data.
- Due to many hotfixes and workarounds over the last months there were quite some parts of code which were not used anymore or which were never. A lot of those parts are removed now in order to make the code clearer and better maintainable.
- The digit and trackler parsing was done in separate classes. I did not see any benefit in that. The actual parsing is not that long. And having it all in the
CruRawReadergives I think a better overview (no need to carry around all different kinds of options and the error reporting can all be done in a single place). - So far the
TrackletHCHeaders were not properly checked. In addition to the wrongly connected links which were not noticed before also a bug in the Sim->Raw conversion was uncovered by the CI (see changes inconstructTrackletHCHeader). This means that at some point we should regenerate the synthetic data samples. Until then a workaround has been put in place, so that the TrackletHCHeaders can still be ignored by the raw reader. This should be activated until the new data samples are available (but only for synthetics, not for real data). - The raw reader now reports for every TF it has seen the total data size of its input (including all headers, so also the RDH), the data which was rejected by the digit and tracklet parsing (here only the words count which could not be read) and the total number of digits and tracklets. Comparing the size of the dropped data to the sizes of digits and tracklets gives an idea on how many digits/tracklets were dropped.
In general I tried to still follow the same logic which was already in place. I think there is still some optimization potential , but the first priority should be to have a stable and maintainable raw data parsing. How stable it is with the changes from this PR still has to be proven in production of course... But my tests look promising.
After this is merged we need to create the CCDB object for the HCID to Link mapping and then add a few changes which are all already prepared in the code. For the synthetic data sample we might need to use the CCDB object as well. Here I need to check which timestamp we are using during running. Maybe we can also keep the default mapping for synthetics.
When going through the data from the noise run 520813 I noticed the current digit parsing was too restrictive. In case there is invalid ADC data discovered I try to find the next MCM header now instead of rejecting the rest of the data from the link completely. Invalid ADC data is present in all half chambers. @bazinski I think it would make sense to add some error counting not only on the level of HCID, but also in the level of individual MCMs. What do you think about it? How hard would it be to add this to QC? Adding it to the event records is very easy and straight forward
Error while checking build/O2/fullCI for ebcf77148712308483a097aa7079b8fc77590737 at 2022-08-30 18:55:
## sw/BUILD/o2checkcode-latest/log
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/Common/test/testGPUsortCUDA.cu:22:10: error: 'boost/test/unit_test.hpp' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUTracking/TRDTracking/GPUTRDTracker.cxx:37:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUTracking/Base/GPUReconstruction.cxx:37:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUTracking/Base/GPUReconstructionCPU.cxx:45:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUTracking/display/GPUDisplay.cxx:36:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDAGenRTC.cu:16:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/cuda/../Shared/Utils.h:26:10: error: 'boost/program_options.hpp' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/cuda/../Shared/Utils.h:26:10: error: 'boost/program_options.hpp' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/Framework/Logger/include/Framework/Logger.h:14:10: error: 'fairlogger/Logger.h' file not found [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:520:12: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:1059:69: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:1175:5: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:1852:16: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:2276:18: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:4336:16: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/sm_20_atomic_functions.h:89:39: error: redefinition of 'atomicAdd' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/Framework/Logger/include/Framework/Logger.h:14:10: error: 'fairlogger/Logger.h' file not found [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:520:12: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:1059:69: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:1175:5: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:1852:16: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:2276:18: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/cuda/std/detail/libcxx/include/type_traits:4336:16: error: CUDA device code does not support variadic functions [clang-diagnostic-error]
/usr/local/cuda-11.7/include/sm_20_atomic_functions.h:89:39: error: redefinition of 'atomicAdd' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/../Shared/Utils.h:146:34: error: use of undeclared identifier 'int4'; did you mean 'int'? [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:64:8: error: unknown type name '__host__' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:64:27: error: expected ';' after top level declarator [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:64:33: error: overloaded 'operator+=' must have at least one parameter of class or enumeration type [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:64:44: error: unknown type name 'int4'; did you mean 'int'? [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:64:53: error: unknown type name 'int4'; did you mean 'int'? [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:66:4: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:66:11: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:67:4: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:67:11: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:68:4: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:68:11: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:69:4: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:69:11: error: member reference base type 'int' is not a structure or union [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:85:1: error: unknown type name '__global__' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:90:19: error: use of undeclared identifier 'blockIdx' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:90:32: error: use of undeclared identifier 'blockDim' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:90:45: error: use of undeclared identifier 'threadIdx' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/Kernels.hip.cxx:90:78: error: use of undeclared identifier 'blockDim' [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/../Shared/Utils.h:146:34: error: use of undeclared identifier 'int4'; did you mean 'int'? [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/GPU/GPUbenchmark/hip/benchmark.hip.cxx:199:35: error: use of undeclared identifier 'int4'; did you mean 'int'? [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/Detectors/EMCAL/calibration/include/EMCALCalibration/EMCALCalibExtractor.h:33:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/Detectors/EMCAL/calibration/include/EMCALCalibration/EMCALCalibExtractor.h:33:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/Detectors/EMCAL/calibration/include/EMCALCalibration/EMCALCalibExtractor.h:33:10: error: 'omp.h' file not found [clang-diagnostic-error]
/sw/SOURCES/O2/9691-slc8_x86-64/0/Detectors/TOF/calibration/src/TOFChannelCalibrator.cxx:23:10: error: 'omp.h' file not found [clang-diagnostic-error]
Full log here.
Error while checking build/O2/o2-dataflow for 0c1a667adfdaf8db386eb794559d25feed2001cc at 2022-08-30 20:13:
## sw/BUILD/O2-latest/log
100% tests passed, 0 tests failed out of 421
86/106 Test #100: test_Framework_test_ParallelProducer ....................***Failed 3.48 sec
[ERROR] pid 24364 (merger) crashed with 1
[ERROR] SEVERE: Device merger (24364) returned with 1
99% tests passed, 1 tests failed out of 101
Full log here.
Error while checking build/O2/o2-cs8 for 0c1a667adfdaf8db386eb794559d25feed2001cc at 2022-08-30 22:58:
## sw/BUILD/O2-latest/log
444/444 Test #400: o2sim_hepmc ..........................................................................***Timeout 400.01 sec
99% tests passed, 1 tests failed out of 444
100% tests passed, 0 tests failed out of 101
Full log here.
@bazinski are the changes here fine with you?
I just let it run over a single TF for a couple of times with the version of the raw reader we have right now and with the one from this PR. With the version in dev:
[57948:trd-datareader]: [14:28:53][INFO] Digits: 36469 (36469 TF), Tracklets: 141789 (141789 TF), DataRead in: 28870.314 MB, Rejected: 34943.840 MB for TF 25651 in 170 ms
New version:
[56579:trd-datareader]: [14:28:03][INFO] Digits: 36233, Tracklets: 141789, DataRead in: 13.470 MB, Rejected: 130.695 kB for TF 25651 in 30 ms
The reported processing time with the changes from this PR is a factor 5.6 faster. Also the reported data sizes are now corrected.
Error while checking build/O2/o2 for 0c1a667adfdaf8db386eb794559d25feed2001cc at 2022-09-01 03:33:
## sw/BUILD/O2-latest/log
100% tests passed, 0 tests failed out of 444
95/106 Test #109: test_Framework_test_SingleDataSource ....................***Failed 3.11 sec
[ERROR] pid 26580 (internal-dpl-injected-dummy-sink) crashed with 1
[ERROR] SEVERE: Device internal-dpl-injected-dummy-sink (26580) returned with 1
99% tests passed, 1 tests failed out of 101
Full log here.
Error while checking build/O2/o2-dataflow-cs8 for 0c1a667adfdaf8db386eb794559d25feed2001cc at 2022-09-01 09:15:
## sw/BUILD/O2-latest/log
100% tests passed, 0 tests failed out of 421
95/106 Test #109: test_Framework_test_SingleDataSource ....................***Failed 3.16 sec
[ERROR] pid 16234 (internal-dpl-injected-dummy-sink) crashed with 1
[ERROR] SEVERE: Device internal-dpl-injected-dummy-sink (16234) returned with 1
99% tests passed, 1 tests failed out of 101
Full log here.
Error while checking build/O2/o2 for 6ff75a5f5a3e2637d89637f41949c009f7f5891b at 2022-09-04 15:47:
## sw/BUILD/O2-latest/log
139/446 Test #91: CCDB/test/testCcdbApi_ConfigParam.cxx ................................................***Failed 18.35 sec
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/GLO/Config/PVertexer/1662298618692/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(130): [1;31;49merror: in "testConfigParamRetrieval": check object != nullptr has failed[0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(132): [1;31;49merror: in "testConfigParamRetrieval": check object->getMemberProvenance("useMeanVertexConstraint") == o2::conf::ConfigurableParam::EParamProvenance::kCCDB has failed[0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(134): [1;31;49merror: in "testConfigParamRetrieval": check p1.getMemberProvenance("useMeanVertexConstraint") == o2::conf::ConfigurableParam::EParamProvenance::kCCDB has failed[0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(135): [1;31;49merror: in "testConfigParamRetrieval": check object == &p1 has failed[0;39;49m
443/446 Test #93: CCDB/test/testBasicCCDBManager.cxx ...................................................***Failed 180.40 sec
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/CachingA/1500/
[FATAL] Got nullptr from CCDB for path Test/CachingA and timestamp 1500
unknown location(0): [4;31;49mfatal error: in "TestBasicCCDBManager": unknown type[0;39;49m
446/446 Test #90: CCDB/test/testCcdbApi.cxx ............................................................***Failed 532.91 sec
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Cannot insert the object in the database</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><hr class="line" /></body></html>[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPath/1662298785435/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(142): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check path2 != nullptr has failed [0 == nullptr][0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(145): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check path2 && path2->getPathString().CompareTo("HelloWorld") == 0 has failed[0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPath/1662298808657/Hello=World/
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPath/1662298822801/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(153): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check md.count("Hello") == 1 has failed [0 != 1][0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(154): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check md["Hello"] == "World" has failed [ != World][0;39;49m
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Cannot insert the object in the database</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><hr class="line" /></body></html>[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/tree2/1662298868240/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(168): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr has failed[0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(169): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && std::strcmp(tree->GetName(), "tree123") == 0 has failed[0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(170): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && tree->GetEntries() == 1 has failed[0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774//1662298869252/
[ERROR] Unable to find object Test/pid16774//1662298869252, Aborting
[ERROR] Local snapshot /tmp/WREg6crI1igg4jVw/Test/pid16774/CCDBPath/snapshot.root not found
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(188): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check snapshot.retrieveFromTFileAny<o2::ccdb::IdPath>(basePath + "CCDBPath", f.metadata) != nullptr has failed[0;39;49m
[ERROR] Local snapshot /tmp/WREg6crI1igg4jVw/Test/pid16774/tree2/snapshot.root not found
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(192): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr has failed[0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(193): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && std::strcmp(tree->GetName(), "tree123") == 0 has failed[0;39;49m
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(194): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && tree->GetEntries() == 1 has failed[0;39;49m
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Cannot insert the object in the database</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><hr class="line" /></body></html>[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1000/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(237): [1;31;49merror: in "timestamptest": check path2 != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1000/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1001/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1002/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1003/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1004/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1005/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1006/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1007/Hello=World/
/sw/SOURCES/O2/9691-slc7_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid16774/CCDBPathUnitTest/1008/Hello=World/
Full log here.
Error while checking build/O2/o2-cs8 for 6ff75a5f5a3e2637d89637f41949c009f7f5891b at 2022-09-04 14:59:
## sw/BUILD/O2-latest/log
443/446 Test #91: CCDB/test/testCcdbApi_ConfigParam.cxx ................................................***Failed 20.05 sec
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/GLO/Config/PVertexer/1662295847856/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(130): [1;31;49merror: in "testConfigParamRetrieval": check object != nullptr has failed[0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(132): [1;31;49merror: in "testConfigParamRetrieval": check object->getMemberProvenance("useMeanVertexConstraint") == o2::conf::ConfigurableParam::EParamProvenance::kCCDB has failed[0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(134): [1;31;49merror: in "testConfigParamRetrieval": check p1.getMemberProvenance("useMeanVertexConstraint") == o2::conf::ConfigurableParam::EParamProvenance::kCCDB has failed[0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi_ConfigParam.cxx(135): [1;31;49merror: in "testConfigParamRetrieval": check object == &p1 has failed[0;39;49m
444/446 Test #93: CCDB/test/testBasicCCDBManager.cxx ...................................................***Failed 91.50 sec
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/CachingA/1500/
[FATAL] Got nullptr from CCDB for path Test/CachingA and timestamp 1500
unknown location(0): [4;31;49mfatal error: in "TestBasicCCDBManager": unknown type[0;39;49m
446/446 Test #90: CCDB/test/testCcdbApi.cxx ............................................................***Failed 434.13 sec
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Cannot insert the object in the database</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><hr class="line" /></body></html>[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPath/1662295997164/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(142): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check path2 != nullptr has failed [0 == nullptr][0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(145): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check path2 && path2->getPathString().CompareTo("HelloWorld") == 0 has failed[0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPath/1662296007265/Hello=World/
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPath/1662296015852/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(153): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check md.count("Hello") == 1 has failed [0 != 1][0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(154): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check md["Hello"] == "World" has failed [ != World][0;39;49m
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Cannot insert the object in the database</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><hr class="line" /></body></html>[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/tree2/1662296037058/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(168): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr has failed[0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(169): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && std::strcmp(tree->GetName(), "tree123") == 0 has failed[0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(170): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && tree->GetEntries() == 1 has failed[0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625//1662296046147/
[ERROR] Unable to find object Test/pid23625//1662296046147, Aborting
[ERROR] Local snapshot /tmp/Zjcz0L6DeSZGRPDP/Test/pid23625/CCDBPath/snapshot.root not found
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(188): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check snapshot.retrieveFromTFileAny<o2::ccdb::IdPath>(basePath + "CCDBPath", f.metadata) != nullptr has failed[0;39;49m
[ERROR] Local snapshot /tmp/Zjcz0L6DeSZGRPDP/Test/pid23625/tree2/snapshot.root not found
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(192): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr has failed[0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(193): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && std::strcmp(tree->GetName(), "tree123") == 0 has failed[0;39;49m
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(194): [1;31;49merror: in "store_retrieve_TMemFile_templated_test": check tree != nullptr && tree->GetEntries() == 1 has failed[0;39;49m
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Cannot insert the object in the database</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><hr class="line" /></body></html>[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1000/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(237): [1;31;49merror: in "timestamptest": check path2 != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1000/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1001/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1002/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1003/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1004/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1005/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1006/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1007/Hello=World/
/sw/SOURCES/O2/9691-slc8_x86-64/0/CCDB/test/testCcdbApi.cxx(242): [1;31;49merror: in "timestamptest": check p != nullptr has failed [0 == nullptr][0;39;49m
[ERROR] Requested resource does not exist: http://ccdb-test.cern.ch:8080/Test/pid23625/CCDBPathUnitTest/1008/Hello=World/
Full log here.
Error while checking build/O2/o2-dataflow-cs8 for 6ff75a5f5a3e2637d89637f41949c009f7f5891b at 2022-09-05 09:42:
## sw/BUILD/O2-latest/log
100% tests passed, 0 tests failed out of 423
100% tests passed, 0 tests failed out of 101
## sw/BUILD/QualityControl-latest/log
31/42 Test #33: multinode_test ..........................***Failed 39.42 sec
2022-09-05 07:41:15.823446 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
2022-09-05 07:41:15.833504 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3919:internal-dpl-clock]: 2022-09-05 07:41:18.146555 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3923:qc-task-TST-MultiNodeRemoteTest52402]: 2022-09-05 07:41:18.164437 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3926:internal-dpl-injected-dummy-sink]: 2022-09-05 07:41:18.164854 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3920:TST-MultiNodeLocalTest35399-proxy]: 2022-09-05 07:41:18.166396 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3925:qc-check-TST-MultiNodeRemoteTest]: 2022-09-05 07:41:18.166556 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3924:qc-check-TST-MultiNodeLocalTest]: 2022-09-05 07:41:18.168169 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3921:sampling2]: 2022-09-05 07:41:18.179219 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3923:qc-task-TST-MultiNodeRemoteTest52402]: 2022-09-05 07:41:18.188529 !!! [1;31mError - Could not find the DPL InfoLogger[0m
[3925:qc-check-TST-MultiNodeRemoteTest]: 2022-09-05 07:41:18.188602 !!! [1;31mError - Could not find the DPL InfoLogger.[0m
[3922:TST-MERGER-MultiNodeLocalTest353991l-0]: 2022-09-05 07:41:18.190922 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3924:qc-check-TST-MultiNodeLocalTest]: 2022-09-05 07:41:18.190623 !!! [1;31mError - Could not find the DPL InfoLogger.[0m
[3934:Dispatcher]: 2022-09-05 07:41:18.203514 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3935:qc-task-TST-MultiNodeLocalTest35399]: 2022-09-05 07:41:18.215356 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3935:qc-task-TST-MultiNodeLocalTest35399]: 2022-09-05 07:41:18.236453 !!! [1;31mError - Could not find the DPL InfoLogger[0m
[3936:sampling2-proxy]: 2022-09-05 07:41:18.215623 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
[3937:TST-MultiNodeLocalTest35399-proxy]: 2022-09-05 07:41:18.202942 !!! [1;31mError - Cycle duration is too short (5), replaced by a duration of 10 seconds.[0m
98% tests passed, 1 tests failed out of 42
Full log here.
Error while checking build/AliceO2/O2/o2/macOS-arm for 6ff75a5f5a3e2637d89637f41949c009f7f5891b at 2022-09-05 14:56:
## sw/BUILD/O2-latest/log
421/445 Test #401: o2sim_hepmc ..........................................................................***Failed Required regular expression not found. Regex=[SIMULATION RETURNED SUCCESFULLY
[ERROR] SHUTTING DOWN DUE TO SIGNALED EXIT IN COMPONENT 77597
Error in <TGeoCompositeShape::MakeNode>: Expression has no boolean operation
Error in <TGeoCompositeShape::ctor>: Composite halfFixationToPipe: cannot parse expression: pipeSupportTubeCarbon
99% tests passed, 1 tests failed out of 445
100% tests passed, 0 tests failed out of 101
Full log here.
Error while checking build/O2/o2-dataflow for 6ff75a5f5a3e2637d89637f41949c009f7f5891b at 2022-09-06 06:25:
## sw/BUILD/O2-latest/log
100% tests passed, 0 tests failed out of 423
95/106 Test #109: test_Framework_test_SingleDataSource ....................***Failed 2.21 sec
[ERROR] pid 12209 (internal-dpl-injected-dummy-sink) crashed with 1
[ERROR] SEVERE: Device internal-dpl-injected-dummy-sink (12209) returned with 1
99% tests passed, 1 tests failed out of 101
Full log here.
A few more words, since this turned out to become a rewrite of most of the actual parsing of the TRD raw data.
- Due to many hotfixes and workarounds over the last months there were quite some parts of code which were not used anymore or which were never. A lot of those parts are removed now in order to make the code clearer and better maintainable.
- The digit and trackler parsing was done in separate classes. I did not see any benefit in that. The actual parsing is not that long. And having it all in the
CruRawReadergives I think a better overview (no need to carry around all different kinds of options and the error reporting can all be done in a single place).- So far the
TrackletHCHeaders were not properly checked. In addition to the wrongly connected links which were not noticed before also a bug in the Sim->Raw conversion was uncovered by the CI (see changes inconstructTrackletHCHeader). This means that at some point we should regenerate the synthetic data samples. Until then a workaround has been put in place, so that the TrackletHCHeaders can still be ignored by the raw reader. This should be activated until the new data samples are available (but only for synthetics, not for real data).- The raw reader now reports for every TF it has seen the total data size of its input (including all headers, so also the RDH), the data which was rejected by the digit and tracklet parsing (here only the words count which could not be read) and the total number of digits and tracklets. Comparing the size of the dropped data to the sizes of digits and tracklets gives an idea on how many digits/tracklets were dropped.
In general I tried to still follow the same logic which was already in place. I think there is still some optimization potential , but the first priority should be to have a stable and maintainable raw data parsing. How stable it is with the changes from this PR still has to be proven in production of course... But my tests look promising.
After this is merged we need to create the CCDB object for the HCID to Link mapping and then add a few changes which are all already prepared in the code. For the synthetic data sample we might need to use the CCDB object as well. Here I need to check which timestamp we are using during running. Maybe we can also keep the default mapping for synthetics.
- agreed.
- of course having the seperate classes is a pain, ergo that would imply a good reason. The code is shared with the compressor, so as not to have 2 pieces of code parsing the respective data. It also permits trivial root macros to parse data.
- not checked as it was not necesarily there so could not be relied on, now it is guaranteed to be there.
- not sure why the numbers went wrong, those are sent to qc.
When going through the data from the noise run 520813 I noticed the current digit parsing was too restrictive. In case there is invalid ADC data discovered I try to find the next MCM header now instead of rejecting the rest of the data from the link completely. Invalid ADC data is present in all half chambers. @bazinski I think it would make sense to add some error counting not only on the level of HCID, but also in the level of individual MCMs. What do you think about it? How hard would it be to add this to QC? Adding it to the event records is very easy and straight forward
It will get complicated as its a simple blob of data sent across to qc, with that granularity it would make sense to rather use a sparse array. It could still be done as a blob as its currently done, just have to change the calculations, and not sure how one syncs that up with qc, similar to the problem with the last error position name having to stay for qc to pass.
of course having the seperate classes is a pain, ergo that would imply a good reason. The code is shared with the compressor, so as not to have 2 pieces of code parsing the respective data. It also permits trivial root macros to parse data.
The parsing methods are public, so nothing is preventing from using them in trivial macros.
not checked as it was not necesarily there so could not be relied on, now it is guaranteed to be there.
Even if they are not always there, when they are, I think they should be checked.