fix: upload non-conflicting files for sharded checkpointing [MD-298]

[azhou-determined]

Ticket

Description

make sharded checkpoint uploads with store_path check for file conflicts across workers before upload.

Test Plan

Checklist

• [ ] Changes have been manually QA'd
• [ ] New features have been approved by the corresponding PM
• [ ] User-facing API changes have the "User-facing API Change" label
• [ ] Release notes have been added as a separate file under docs/release-notes/ See Release Note for details.
• [ ] Licenses have been included for new code which was copied and/or modified from any external code

[netlify[bot]]

✅ Deploy Preview for determined-ui ready!

Name	Link
🔨 Latest commit	93f186338ea7b98f77eff72a62b047be25825175
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/669080630b56410008c86ad4
😎 Deploy Preview	https://deploy-preview-9598--determined-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 53.02%. Comparing base (7fab87b) to head (93f1863).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #9598   +/-   ##
=======================================
  Coverage   53.01%   53.02%           
=======================================
  Files        1255     1255           
  Lines      152884   152960   +76     
  Branches     3233     3234    +1     
=======================================
+ Hits        81053    81106   +53     
- Misses      71680    71703   +23     
  Partials      151      151           

Flag	Coverage Δ
backend	44.19% <ø> (-0.05%)	:arrow_down:
harness	72.85% <100.00%> (+0.08%)	:arrow_up:
web	51.37% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
harness/determined/common/storage/base.py	93.38% <100.00%> (ø)
harness/determined/common/storage/cloud.py	100.00% <100.00%> (ø)
harness/determined/common/storage/shared.py	82.22% <100.00%> (ø)
harness/determined/core/_checkpoint.py	95.45% <100.00%> (+0.99%)	:arrow_up:
harness/tests/core/test_checkpoint.py	100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes