TPC Timeseries: Adding cov and delta track params at vertex

[matthias-kleiner]

[github-actions[bot]]

REQUEST FOR PRODUCTION RELEASES: To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available async-2023-pbpb-apass async-2023-pp-apass1 async-2022-pp-apass6 async-2022-pp-apass4 async-mc async-data

[miranov25]

[miranov25]

Hello @matthias-kleiner

As we discused in today meeting, we would like to monitor also TRD/TOF matching efficiency. While the TOF we can get from deltas for the TRD we do not have yet anything. We should prepare one byte for the TRD tracklets - getTrackleIndex to be converted to the bitmask one bit per layer -in case tracklet is assigned at layer .

[miranov25]

Posting comment from Mattermost:

Hi @shahoian, @mkleiner, and @wiechula,

I am writing to discuss some results related to our covariance matrix validation using the new code for the angle delta. This is now being used for edge cut validation optimization (Matthias pull request - https://github.com/AliceO2Group/AliceO2/pull/12928)

While the problem may lie in the current implementation in the pull request, Jens' recent test runs have shown unexpected behaviour. We observe pulls close to 1 (|qpt| < 2), as it should be, but we also see a strong angular bias (linear with q/pt) - see slide below

The pull request implementation is a possible culprit, but I wanted to bring it to your attention anyway. I also suggested @wiechula to continue the full statistical tests so I can extract more different information. This would help us isolate the cause of the problem, be it reconstruction, alignment, or something else.

Has anyone else run similar tests on angle delta and angle draws?

My desire for this particular data set stems from the observed improvements in edge cut matching efficiency. I wanted to measure the combined track performance benchmark using angular pulls and angular residuals to better assess performance in terms of systematic error and handling of edge clusters before suggesting a choice.

Best regards,

Marian

[miranov25]

Looks like z vertex is not used in the update - is mean vertex used?

[shahor02]

@miranov25 @matthias-kleiner see my reply in https://mattermost.web.cern.ch/alitpccomact/pl/o1c8rjbprtf7jri4fh36e5quuc

[miranov25]

Updating the TPC using "real" vertex instead of the mean vertex would be preferable. However, I'm uncertain about the ease of implementation and whether a collision vertex is available at that moment. The spread of the vertex mostly in the Z direction is too significant to overlook.

[miranov25]

Hi @matthias-kleiner,

The changes introduced in the commit above will most probably address the local X-match issue described in https://github.com/AliceO2Group/AliceO2/pull/12928#issuecomment-2023421138.

As discussed previously (here: https://github.com/AliceO2Group/AliceO2/pull/12928#issuecomment-2023470047), it appears the "mean" vertex position is currently being used for the constrained track update. This approach can be problematic, particularly for the z-coordinate.

Could you consider using the assigned vertex position instead? This would likely improve the accuracy of the update.

Thanks, Marian

[miranov25]

Covariance matrix for qpt- source code to be checked

• is the "overoptimistic" covariance matrix only in our skimming code or is the systematic error not use?

[miranov25]

We've identified a persistent issue with the incorrect track covariance for TPCITS tracks in the skimmed data.

To simplify debugging and further studies, I propose storing the following full tracks in the skimmed data:

• TPC track at inner wall (22 floats)
• TPC constrained track
• TPC+ITS track at innermost point (similar to AO2D)

We understand that this will increase the data volume. To mitigate this, we can consider:

Dumping full tracks only for a representative subset of the skimmed data. For the remaining data, we could utilize 0-pointers if feasible. If using 0-pointers is not possible, we might need to explore slightly less aggressive skimming.. This approach should simplify debugging the covariance issue and facilitate further analysis of TPCITS tracks while minimizing the impact on data storage requirements.

[shahor02]

Empty pointers are not supported by the streamer. But I don't understand why do you need to increase the output of the central production just to debug the code which should produce a more compact output. Why don't you produce debug data locally and debug on it?

[miranov25]

Hello @shahor02 , @matthias-kleiner , @wiechula and all.

The data volume for track parameters is substantial, with 88 bytes allocated for 4 parameters. To enhance storage efficiency, I've suggested a more significant downsampling factor. Could you inform me if there exists a compression class—encompassing relative/absolute precision and correlation—outside of AO2D that we could leverage for additional compression? If such a class doesn't exist, I'm prepared to develop one. Interestingly, we can also compress one track in relation to another as I had shown before (dEdx storage in respect to the average dEdx in the trackQA table)

Although track parameters and covariance cannot be directly derived from existing data, having comprehensive track information is vital for our performance reports. Both covariance and correlation are influenced by systematic errors and the track's trajectory, including impacts from dead zones.

Our past experiences underscore the perils of depending exclusively on local production for high-statistics performance evaluations. Both local and GSI-based tests suffer from limited statistical reach. We must acknowledge the challenges we face in data comprehension during Run 3 and the feedback loop for analysts. Recognizing these issues is the first step towards facilitating the work of our experts.

During Runs 1 and 2, the use of skimmed data significantly eased the process of conducting checks, ensuring that track performance—including tracking and PID—remained robust across all momentum ranges. This approach allowed us to quickly sift through and analyze production-equivalent data to pinpoint potential issues, proving invaluable for identifying and resolving past challenges. I must emphasize that without the availability of this skimmed data and format, pinpointing and remedying certain issues would have been unfeasible.

Modified Proposal:

• Downsample track parameters: Implement downsampling to reduce data volume further
• Separate storage for track parameters: Store downsampled track parameters in a separate tree (e.g., a dedicated tree with indices) for efficient access during performance checks.

Technical Considerations:

• Since we're already using skimmed data with a sampling rate of O(10^-3), downsampling the track parameters further is technically feasible.
• Storing zero-pointers might not be possible. In that case, we can explore these alternatives:
  • Check for "default 0" track content (likely well-compressed).
  • Store the data in a separate tree with unique indices to enable efficient navigation in both directions.

Suggested Options for streaming:

• kSaveTracks (default streamer): Enables saving tracks for local checks.
• kSaveTrackFraction (production): Stores downsampled data in a separate tree format for production usage.

This approach provides flexibility and allows us to optimize storage while maintaining the ability to conduct thorough performance checks.

[shahor02]

@miranov25 there is no AOD-like track param compression. My comment was that we should not store extra stuff in production just to debug skimming code.

[miranov25]

Hello @shahor02 .

Thank you for the clarification.

Based on your input, I propose we first proceed with the options as I previously outlined. If I successfully implement relative track compression in reference to the track, we can adopt Motion 1 (kSaveTracks). Should this not be implemented, we can default to Motion 2 (kSaveTrackFraction).

Ideally, these changes would be in place before the major production phase. Unfortunately, we currently lack these capabilities in our ongoing production, which are essential for differentially controlling performance—specifically for monitoring pT resolution at very high momenta (from 5-10 GeV up to 100 GeV).

[miranov25]

TPC ITS chi2 is missing in the time series - will be preferable to add it

add TPC ITS chi2 - TODO - missing in the time series - resent only in skimmed data

[matthias-kleiner]

Hello @miranov25 , did you check if the new variables are fine, so that we can merge this? Or were there some problems?

[miranov25]

Hello @miranov25 , did you check if the new variables are fine, so that we can merge this? Or were there some problems?

Yes, please proceed with the merge. The TPC-ITS chi2 that I mentioned earlier is included. I overlooked it initially.

The option for the full parameter dump has not been implemented yet but can be addressed in the next pull request.

Indeed, obtaining a subset of the data with these additions will be crucial, as we discussed three weeks ago.

[miranov25]

Hello @matthias-kleiner

Preparing the combined tracking resolution and bias parameterization I had seen that the wrong vertex is used to fill the content.

I am currently reviewing the source code and will also update it on GitHub:

The DCA is calculated and stored only for the reference point (0,0). This is handled in the following code snippet:

if (propagator->propagateToDCA(refPoint, trackITSTPCTmp, propagator->getNominalBz(), mFineStep, mMatType, &dcaITSTPC))

https://github.com/matthias-kleiner/AliceO2/blob/456328b5c539f5fafe394c215a3eff9b03f3398f/Detectors/TPC/workflow/src/TPCTimeSeriesSpec.cxx#L1191-L1193

• later it is calculated in respect to the real vertex, but it is added only to the tmp which is not used later: https://github.com/matthias-kleiner/AliceO2/blob/456328b5c539f5fafe394c215a3eff9b03f3398f/Detectors/TPC/workflow/src/TPCTimeSeriesSpec.cxx#L1200C12-L1202

Should I fix it or can we meet and prepare the new pull request

[matthias-kleiner]

Hello @matthias-kleiner

Preparing the combined tracking resolution and bias parameterization I had seen that the wrong vertex is used to fill the content.

I am currently reviewing the source code and will also update it on GitHub:

The DCA is calculated and stored only for the reference point (0,0). This is handled in the following code snippet:

if (propagator->propagateToDCA(refPoint, trackITSTPCTmp, propagator->getNominalBz(), mFineStep, mMatType, &dcaITSTPC))

https://github.com/matthias-kleiner/AliceO2/blob/456328b5c539f5fafe394c215a3eff9b03f3398f/Detectors/TPC/workflow/src/TPCTimeSeriesSpec.cxx#L1191-L1193

* later it is calculated in respect to the real vertex, but it is added only to the tmp which is not used later:
  https://github.com/matthias-kleiner/AliceO2/blob/456328b5c539f5fafe394c215a3eff9b03f3398f/Detectors/TPC/workflow/src/TPCTimeSeriesSpec.cxx#L1200C12-L1202

Should I fix it or can we meet and prepare the new pull request

Hello, thanks for spotting. If you want you can make the PR. I think the changes are trivial.

[miranov25]

Hallo @matthias-kleiner,

I'm encountering a systematic build error with O2 since updating my macOS. The error message indicates an incompatibility between the generated code and the LLVM version used during linking.

[ 7%] Generating ../irhelpers.bc
/opt/homebrew/opt/llvm@14/bin/llvm-link: /Users/miranov25/alicesw/sw/BUILD/15ca03faaeb56df32a6601dc5e3160fb279ce409/arrow/src/gandiva/precompiled/arithmetic_ops.bc: error: Opaque pointers are only supported in -opaque-pointers mode (Producer: 'APPLE_1_1500.1.0.2.5_0' Reader: 'LLVM 14.0.6')
/opt/homebrew/opt/llvm@14/bin/llvm-link: error: loading file '/Users/miranov25/alicesw/sw/BUILD/15ca03faaeb56df32a6601dc5e3160fb279ce409/arrow/src/gandiva/precompiled/arithmetic_ops.bc'
make[2]: *** [src/gandiva/irhelpers.bc] Error 1
make[1]: *** [src/gandiva/precompiled/CMakeFiles/precompiled.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

I'm working on troubleshooting the issue with my local installation. However, without the ability to run local tests, I'm hesitant to commit any changes at this time.

Given the upcoming MC and raw data tests, I was wondering if we could collaborate on a potential solution, perhaps together with Jens.

Thanks, Marian