feat(api): Pause run after recoverable errors

[SyntaxColoring]

Steps towards EXEC-301.

Overview

Test Plan

Changelog

Review requests

Risk assessment

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 67.34%. Comparing base (583dcf6) to head (4221574). Report is 129 commits behind head on edge.

Additional details and impacted files

@@            Coverage Diff             @@
##             edge   #14646      +/-   ##
==========================================
- Coverage   67.56%   67.34%   -0.23%     
==========================================
  Files        2521     2485      -36     
  Lines       72251    71342     -909     
  Branches     9311     9014     -297     
==========================================
- Hits        48815    48042     -773     
+ Misses      21240    21157      -83     
+ Partials     2196     2143      -53     

Flag	Coverage Δ
g-code-testing	92.43% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
api/src/opentrons/ordered_set.py	100.00% <ø> (ø)
...i/src/opentrons/protocol_engine/actions/actions.py	100.00% <ø> (ø)
...rons/protocol_engine/execution/command_executor.py	100.00% <ø> (ø)
...s/protocol_engine/execution/create_queue_worker.py	100.00% <ø> (ø)
...i/src/opentrons/protocol_engine/protocol_engine.py	100.00% <ø> (ø)
...pi/src/opentrons/protocol_engine/state/commands.py	99.36% <ø> (-0.09%)	:arrow_down:
...opentrons/protocol_runner/legacy_command_mapper.py	98.33% <ø> (ø)

... and 220 files with indirect coverage changes

[SyntaxColoring]

I've run through the test plan in the PR description, and also run a couple of other random protocols as a smoke test to make sure this didn't break anything else.