Roundoff error causes stuck tracks at large Z planes with grazing angles

[sethrj]

Failing track in ATLAS (using 10k max steps, CELER_NONFATAL_FLUSH=1) and

export ATHENA_CORE_NUMBER=2
export CELER_DISABLE_DEVICE=1
export CELER_LOG=debug
export CELER_LOG_LOCAL=debug

Sim_tf.py \
    --CA \
    --multithreaded \
    --detectors 'Calo' \
    --conditionsTag 'default:OFLCOND-MC21-SDR-RUN4-02' \
    --geometryVersion 'default:ATLAS-P2-RUN4-04-00-00' \
    --preInclude 'EVNTtoHITS:Campaigns.PhaseIISimulation' \
    --simulator 'FullG4MT' \
    --physicsList 'FTFP_BERT_ATL_Celer' \
    --postInclude 'PyJobTransforms.UseFrontier' \
    --preInclude 'EVNTtoHITS:Campaigns.MC23eSimulationMultipleIoV,SimulationConfig.disablePhotonRussianRoulette,SimulationConfig.disableNeutronRussianRoulette,SimulationConfig.disableFrozenShowersFCalOnly' \
    --preExec 'flags.GeoModel.EMECStandard=True;flags.Sim.OptionalUserActionList += ["G4UserActions.G4UserActionsConfig.CelerOffloadToolCfg"]' \
    --postExec 'cfg.getService("StandardField").UseSoleCurrent=0.;cfg.getService("StandardField").UseToroCurrent=0.' \
    --inputEVNTFile "/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/CampaignInputs/mc21/EVNT/mc21_13p6TeV.601229.PhPy8EG_A14_ttbar_hdamp258p75_SingleLep.evgen.EVNT.e8453/EVNT.29328277._003902.pool.root.1" \
    --outputHITSFile "test.CA.HITS.pool.root" \
    --maxEvents 4 \
    --jobNumber 1 \
    --postExec 'with open("ConfigSimCA.pkl", "wb") as f: cfg.store(f)' \
    --imf False

The failing track has:

{
  "geo": {
    "dir": [
      0.5784236876658104,
      0.8157365000698582,
      -9.290358099212079E-7
    ],
    "is_on_boundary": true,
    "is_outside": false,
    "pos": [
      [
        240.97769534015998,
        179.56803215217408,
        3444.5
      ],
      "mm"
    ],
    "volume_id": "HGTD::HGTDSupportPlate"
  },
  "mat": "Aluminium_Ma",
  "particle": {
    "energy": [
      1.0930431167150243,
      "MeV"
    ],
    "particle_id": "e-"
  },
  "sim": {
    "event_id": 0,
    "num_steps": 9735,
    "parent_id": 537399,
    "post_step_action": "tracking-cut",
    "status": "errored",
    "step_length": [
      1E-8,
      "mm"
    ],
    "time": [
      11.880573899835621,
      "ns"
    ],
    "track_id": 609169
  },
  "thread_id": 12874,
  "track_slot_id": 12874
}

• The returned step of 1e-8 is coming from static constexpr Precision kBoundaryPush = 10 * vecgeom::kTolerance in BVHNavigator
• With a step of 1e-8, a Z direction component of 1e-7, and a position of ~1e3, the change in Z is less than 1 ULP, so the Z position doesn't change at all at the step, so it doesn't numerically exit the Z plane
• When the bumped position is tested the local point's Z coordinate doesn't change, so it doesn't see the new volume

The failing shape is:

    <tube aunit="deg" deltaphi="360" lunit="mm" name="HGTDSupportPlate" rmax="660" rmin="120" startphi="0" z="1"/>

as part of HGTD::HGTDSupportPlate, which is placed in a few locations (z=6927.5,6934.5,6952.5,6959.5 and z=5.5,...,37.5) inside of HGTD::HGTD (z=+3482.5) inside ITK::ITK inside Atlas::Atlas.

[sethrj]

Related: might be fixed by improved navigation with #2111