HPCC-31760 Protect against delta save failures

[jakesmith]

If there are temporary failures (e.g. failures to write to disk), then protect against the delta writer thread from exiting. Retry all disk actions. NB: the initial inspiration for this was that there was a failure due to Dali running out of file handles. Which itself is probably caused by the flood of connections from AKS VM's to Dali.

Type of change:

• [ ] This change is a bug fix (non-breaking change which fixes an issue).
• [ ] This change is a new feature (non-breaking change which adds functionality).
• [x] This change improves the code (refactor or other change that does not change the functionality)
• [ ] This change fixes warnings (the fix does not alter the functionality or the generated code)
• [ ] This change is a breaking change (fix or feature that will cause existing behavior to change).
• [ ] This change alters the query API (existing queries will have to be recompiled)

Checklist:

• [x] My code follows the code style of this project.
  • [ ] My code does not create any new warnings from compiler, build system, or lint.
• [x] The commit message is properly formatted and free of typos.
  • [ ] The commit message title makes sense in a changelog, by itself.
  • [ ] The commit is signed.
• [ ] My change requires a change to the documentation.
  • [ ] I have updated the documentation accordingly, or...
  • [ ] I have created a JIRA ticket to update the documentation.
  • [ ] Any new interfaces or exported functions are appropriately commented.
• [x] I have read the CONTRIBUTORS document.
• [x] The change has been fully tested:
  • [ ] I have added tests to cover my changes.
  • [ ] All new and existing tests passed.
  • [ ] I have checked that this change does not introduce memory leaks.
  • [ ] I have used Valgrind or similar tools to check for potential issues.
• [x] I have given due consideration to all of the following potential concerns:
  • [ ] Scalability
  • [ ] Performance
  • [ ] Security
  • [ ] Thread-safety
  • [ ] Cloud-compatibility
  • [ ] Premature optimization
  • [ ] Existing deployed queries will not be broken
  • [ ] This change fixes the problem, not just the symptom
  • [ ] The target branch of this pull request is appropriate for such a change.
• [x] There are no similar instances of the same problem that should be addressed
  • [ ] I have addressed them here
  • [ ] I have raised JIRA issues to address them separately
• [ ] This is a user interface / front-end modification
  • [ ] I have tested my changes in multiple modern browsers
  • [ ] The component(s) render as expected

Smoketest:

• [ ] Send notifications about my Pull Request position in Smoketest queue.
• [ ] Test my draft Pull Request.

Testing:

[github-actions[bot]]

Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-31760

Jirabot Action Result: Workflow Transition: Merge Pending Updated PR Assigning user: [email protected]

[jakesmith]

@ghalliday @mckellyln - please review NB: quite a lot of the apparent change is moving code inside try/catch blocks.

[jakesmith]

@ghalliday - please see commit #2

[jakesmith]

@ghalliday - squashed.