Write manifests atomically

[twz123]

Description

Part of #1814.

Use a temporary-write-and-rename approach to write manifests to disk, so that the appliers never see partially written files. Google's renameio library does this by creating temorary files whose names start with a dot. So let the appliers ignore anything that starts with a dot. That way, there's also no more need to ignore chmod events. They actually can matter, when files become readable or unreadable for the current process.

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation update

How Has This Been Tested?

• [x] Manual test
• [ ] Auto test added

Checklist:

• [x] My code follows the style guidelines of this project
• [x] My commit messages are signed-off
• [x] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] My changes generate no new warnings
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] New and existing unit tests pass locally with my changes
• [ ] Any dependent changes have been merged and published in downstream modules
• [ ] I have checked my code and corrected any misspellings

[twz123]

(It is unclear to me why GitHub isn't removing already merged commits from a PR, but enforces a rebase.)

[twz123]

Turned into draft. Will revisit that, but not right now.

[github-actions[bot]]

The PR is marked as stale since no activity has been recorded in 30 days

[github-actions[bot]]

The PR is marked as stale since no activity has been recorded in 30 days

[twz123]

@jnummelin Revisited this one. Removed the renameio lib as it was a bit awkward to use with its strong emphasis on "this is posix only". Doing this ourselves allows us to implement a different naming scheme for the temp files, getting rid of all that dotted prefix complexity. Renameio has quite some knobs, but none for the temporary file name pattern.

[s0j]

https://github.com/natefinch/atomic does something similar, worth using? The only drawback I see is that you can't explicitly specify the file mode.

[twz123]

https://github.com/natefinch/atomic does something similar, worth using? The only drawback I see is that you can't explicitly specify the file mode.

That's interesting. That lib uses MoveFileEx with MOVEFILE_REPLACE_EXISTING|MOVEFILE_WRITE_THROUGH. Golang's standard lib does the same, except that it calls it only with MOVEFILE_REPLACE_EXISTING. They claim that this is not atomic, and there seems to be no way to achieve this on Windows.

The docs on the write through flag say:

The function does not return until the file is actually moved on the disk. Setting this value guarantees that a move performed as a copy and delete operation is flushed to disk before the function returns. The flush occurs at the end of the copy operation.

So I doubt that this gives much extra benefit.

I already experimented with renameio in this PR, and I eventually concluded that it's easiest to do this on our own. In the end, this gives us all the possibilities, i.e. influence where the temp files are stored and how they are named, change file permissions and so on. Both libs don't offer all of those knobs.