Add full-capability G4Track/Step reconstruction

[sethrj]

Create a data structure that contains all the attributes needed to fully reconstruct a G4Track for sending to Geant4 via G4StackManager::PushOneTrack (#1894), or G4VSensitiveDetector::Hit (#581, #1397), or G4VProcess::PostStepDoIt (#1898).

Required for:

• Improved hit processing (track data over entire step): EM hits #581 and optical #1397
• Extra physics (track data over most of a step): #1898
• Region-based return to Geant4 (track state at end of step): #2084
• Potential "tail" execution when little work remains in an event [^1]: apt-sim/AdePT#360
• Stepping action (track data over entire step): #1745

[^1]: We usually think of this in terms of "few tracks in the tracking loop" or "many steps taken" which are proxies for "small total amount of energy" which is a proxy for "not much work left".

Plan:

• Define step point data structure with all necessary data: maybe could just reuse StepStateDataImpl in celeritas/user/StepData ?
• Defer volume instance stack reconstruction until ORANGE capability is in place and path-to-scalar (i.e., volume instance stack to UniqueVolumeInstanceId) mapping is set up
• Move reconstruction parts from HitProcessor etc. into new classes for reconstructing StepPoint, Track, and Step:
  • Track reconstructor emits a unique_ptr<G4Track> that can be stacked or used as a "per-particle-type" object for detector callbacks
  • Touchable updater gets moved into step point converter
• Identify additional attributes that need to be copied from GPU

Implementation details:

• Collection groups: step state, persistent state, track state
  • Two step states and one persistent state live in track state
  • Step state has position, direction, volume, etc.: key track attributes
  • Persistent state has (particle ID? primary ID? step counters?)
  • Do not worry about making data configurable like we do for hit processor: assume we copy everything
• Managing the data:
  • Before pre-step, swap pre- and post- array pointers [^2], and then record newly initialized particles into the array
  • Also at pre-step, record persistent state from newly initialized particles
  • Async copy out of persistent and pre-step immediately after kernel
  • At post-step, record all updated states before track is destroyed/replaced by secondary
  • Async copy out of post-step state
• Helper classes to convert step points, persistent state, etc.

[^2]: Not sure if we can do this with "swap" because that introduces a CPU -> GPU synchronization; we might need to pass in the step iteration count and do auto& pre_step_data = step_data[steps % 2]; post.. = step-data[(steps + 1) % 2].

• New structural concept: a group of actions (kernels) that have different ActionStepOrder but share the same aux data (we do this in a number of places already and will do it in more); this will let us coordinate hit processing asynchronously

[esseivaju]

for sending to Geant4 via G4StackManager::PushOneTrack

If we push a track back to Geant4 through the G4StackManager, when it's about to start tracking it, wouldn't it offload it to our TrackingManager again?

[sethrj]

Yeah this is something I forgot about/was wondering: I think AdePT has to implement their own Geant4 stepping loop (maybe even using g4hepem rather than regular g4 physics), but perhaps there's a better way. In G4EventManager::DoProcessing is where it checks for new tracks having custom tracking managers; but the "regular" tracking looks to be essentially:

        tm->ProcessOneTrack( track );
        sm->PushOneTrack( track, aTrajectory );
        em->StackTracks( trackManager->GimmeSecondaries() );

where em is the G4EventManager singleton, tm = em->GetTrackingManager(), and sm = em->GetStackManager().

So perhaps our custom tracking manager could just call those if we've flagged the track (pointer in a set?) as being managed byGeant4?

[esseivaju]

We could get the G4TrackingManager and push the track to it directly which would work but there are inconvenient with this approach:

• Bypass the stacking manager, i.e., no callback to user stacking action
• The tracking manager will transport the track to completion, I don't think there is a hook to get the track back onto the GPU

My understanding is that AdePT has two tracking manager, using either AdePT or G4HepEM for transport and a track can be exchanged between these two tracking manager depending on the region. I am not sure if G4HepEM will handle fast simulation processes or if it will just use its own EM processes.

[esseivaju]

Regarding the work tracked by this issue, are we still missing required data when reconstructing tracks/steps? Now that we reconstruct primary/parent ID, user infos, process, touchable, I think this can be closed.

[sethrj]

@esseivaju I see that the track ID is set but not the parent ID (does that matter?) Also we aren't yet mapping the "hit" process to a Geant4 process: we're propagating the process that created the secondary I think.

And yeah can you create a separate issue for "onloading" to discuss the event/tracking/stacking managers?

[amandalund]

Do we need to reconstruct Geant4 tracks with unique track IDs (and associated parent IDs) for anything, or is propagating the original offloaded track's IDs tp all of its descendants enough for these use cases?

[esseivaju]

Good question, we track Celeritas' parentID for secondaries but when returning hits to Geant4 SDs that were produced by Celeritas secondaries, the associated track is the original offloaded track with its original properties.

[sethrj]

This means we'd be calling "Hit" with a track ID long after the PostUserTrackingAction is called. That might be a problem for MC truth implementations...

[esseivaju]

It's not obvious how we'd reconstruct G4Tracks for Celeritas secondaries, we don't know which track ids are already used by Geant4.

[sethrj]

1. Step gather and aux for optical photons @Rashika-Gupta ; (@sethrj help with generalizing this)
2. Gather all step info (not just for start detector) for main core loop for EM extra physics and stepping action and region onload: needed for #1898 ; adding new class to substitute in main loop for StepCollector
3. Region-based onload: @esseivaju
4. Simultaneously with 3, calling back with step data to G4 hadronics @whokion
5. Optimize: swap pre/post, compacting, buffering @amandalund @esseivaju (possibly @LSchwiebert ); defer decision to refactor StepData to here

[sethrj]

For step 1 with @Rashika-Gupta :

• template<Ownership, MemSpace> struct StepStateData has pre- and post-step states
• template<MemSpace M> using StepState = AuxStateData<StepStateData, M>; is the aux data interface
• class StepParams : public AuxParamsInterface, ..., constructed with SDParams and AuxId, creates step state data and provides access with
• template<StepPoint P> struct StepGatherExecutor (in optical namespace) has a NativeRef<StepStateData> and will copy data from optical core track to the step data. The native ref can be obtained via step_params_->ref<MemSpace::native>(*state.aux());
• template<StepPoint P> class StepGatherAction (in optical namespace) is constructed with shared_ptr<StepParams> and launches StepGatherExecutor
• Construct all those in the optical core params